Loading dylibs

[Vishwas Siravara]

Hi guys,

I have a flink application that loads a dylib like this 

System.loadLibrary("vibesimplejava");

The application runs fine , when I restart the job I get this exception : 

com.visa.aip.cryptolib.aipcyptoclient.EncryptionException: Unexpected errorjava.lang.UnsatisfiedLinkError: Native Library /usr/mware/SimpleAPI/voltage-simple-api-java-05.12.0000-Linux-x86_64-64b-r234867/lib/libvibesimplejava.so already loaded in another classloader

This happens because the dylib has already been loaded once by the taskmanger, how can I mitigate this? It seems problematic if two applications are loading the same dylib. 

Thanks,

Vishwas

[Jörn Franke]

I don’t know Dylibs in detail, but can you call a static method where it checks if it has been already executed and if not then it loads the library (Singleton pattern)?

Am 27.08.2019 um 06:39 schrieb Vishwas Siravara <[hidden email]>:

Hi guys,

I have a flink application that loads a dylib like this 

System.loadLibrary("vibesimplejava");

The application runs fine , when I restart the job I get this exception : 

com.visa.aip.cryptolib.aipcyptoclient.EncryptionException: Unexpected errorjava.lang.UnsatisfiedLinkError: Native Library /usr/mware/SimpleAPI/voltage-simple-api-java-05.12.0000-Linux-x86_64-64b-r234867/lib/libvibesimplejava.so already loaded in another classloader

This happens because the dylib has already been loaded once by the taskmanger, how can I mitigate this? It seems problematic if two applications are loading the same dylib. 

Thanks,

Vishwas

[Vishwas Siravara]

Hi Jörn, 
I tried that. Here is my snippet : 

String[] loadedlibs =  getLoadedLibraries(Thread.currentThread().getContextClassLoader());
if(!containsVibeSimpleLib(loadedlibs)) {
    System.loadLibrary("vibesimplejava");
}

Now I get the exception Unexpected errorjava.lang.UnsatisfiedLinkError: com.voltage.securedata.enterprise.ConstantsNative.DIGEST_MD5()I which means that it could not find vibesimplejava in the loaded libs but I know that the if was not executed because vibesimplejava was present in loadedlibs( the control never went inside the if block. Any other suggestions?

Thanks,

Vishwas

On Tue, Aug 27, 2019 at 12:25 AM Jörn Franke <[hidden email]> wrote:

I don’t know Dylibs in detail, but can you call a static method where it checks if it has been already executed and if not then it loads the library (Singleton pattern)?

Am 27.08.2019 um 06:39 schrieb Vishwas Siravara <[hidden email]>:

Hi guys,

I have a flink application that loads a dylib like this 

System.loadLibrary("vibesimplejava");

The application runs fine , when I restart the job I get this exception : 

com.visa.aip.cryptolib.aipcyptoclient.EncryptionException: Unexpected errorjava.lang.UnsatisfiedLinkError: Native Library /usr/mware/SimpleAPI/voltage-simple-api-java-05.12.0000-Linux-x86_64-64b-r234867/lib/libvibesimplejava.so already loaded in another classloader

This happens because the dylib has already been loaded once by the taskmanger, how can I mitigate this? It seems problematic if two applications are loading the same dylib. 

Thanks,

Vishwas

[Aleksey Pak]

Hi Vishwas,

There is a known issue in the Flink Jira project [1].

Is it possible that you have encountered the same problem?

[1]: https://issues.apache.org/jira/browse/FLINK-11402

Regards,

Aleksey

On Tue, Aug 27, 2019 at 8:03 AM Vishwas Siravara <[hidden email]> wrote:

Hi Jörn, 
I tried that. Here is my snippet : 

String[] loadedlibs =  getLoadedLibraries(Thread.currentThread().getContextClassLoader());
if(!containsVibeSimpleLib(loadedlibs)) {
    System.loadLibrary("vibesimplejava");
}

Now I get the exception Unexpected errorjava.lang.UnsatisfiedLinkError: com.voltage.securedata.enterprise.ConstantsNative.DIGEST_MD5()I which means that it could not find vibesimplejava in the loaded libs but I know that the if was not executed because vibesimplejava was present in loadedlibs( the control never went inside the if block. Any other suggestions?

Thanks,

Vishwas

On Tue, Aug 27, 2019 at 12:25 AM Jörn Franke <[hidden email]> wrote:

I don’t know Dylibs in detail, but can you call a static method where it checks if it has been already executed and if not then it loads the library (Singleton pattern)?

Am 27.08.2019 um 06:39 schrieb Vishwas Siravara <[hidden email]>:

Hi guys,

I have a flink application that loads a dylib like this 

System.loadLibrary("vibesimplejava");

The application runs fine , when I restart the job I get this exception : 

com.visa.aip.cryptolib.aipcyptoclient.EncryptionException: Unexpected errorjava.lang.UnsatisfiedLinkError: Native Library /usr/mware/SimpleAPI/voltage-simple-api-java-05.12.0000-Linux-x86_64-64b-r234867/lib/libvibesimplejava.so already loaded in another classloader

This happens because the dylib has already been loaded once by the taskmanger, how can I mitigate this? It seems problematic if two applications are loading the same dylib. 

Thanks,

Vishwas

[Vishwas Siravara]

Yes this is exactly what happens , as a work around I created a small jar file which has code to load the dylib and I placed it under the lib folder , this library is in provided scope in my actual job, so the dylib gets loaded only once when the tm/jm jvm starts . 

What I found interesting in my old approach was even when I check whether the dylib has already been loaded in the current thread , and if it is I still get the unsatisfied link error even though that dylib is loaded in the task manager . 

On Wed, Aug 28, 2019 at 7:04 AM Aleksey Pak <[hidden email]> wrote:

Hi Vishwas,

There is a known issue in the Flink Jira project [1].

Is it possible that you have encountered the same problem?

[1]: https://issues.apache.org/jira/browse/FLINK-11402

Regards,

Aleksey

On Tue, Aug 27, 2019 at 8:03 AM Vishwas Siravara <[hidden email]> wrote:

Hi Jörn, 
I tried that. Here is my snippet : 

String[] loadedlibs =  getLoadedLibraries(Thread.currentThread().getContextClassLoader());
if(!containsVibeSimpleLib(loadedlibs)) {
    System.loadLibrary("vibesimplejava");
}

Now I get the exception Unexpected errorjava.lang.UnsatisfiedLinkError: com.voltage.securedata.enterprise.ConstantsNative.DIGEST_MD5()I which means that it could not find vibesimplejava in the loaded libs but I know that the if was not executed because vibesimplejava was present in loadedlibs( the control never went inside the if block. Any other suggestions?

Thanks,

Vishwas

On Tue, Aug 27, 2019 at 12:25 AM Jörn Franke <[hidden email]> wrote:

I don’t know Dylibs in detail, but can you call a static method where it checks if it has been already executed and if not then it loads the library (Singleton pattern)?

Am 27.08.2019 um 06:39 schrieb Vishwas Siravara <[hidden email]>:

Hi guys,

I have a flink application that loads a dylib like this 

System.loadLibrary("vibesimplejava");

The application runs fine , when I restart the job I get this exception : 

com.visa.aip.cryptolib.aipcyptoclient.EncryptionException: Unexpected errorjava.lang.UnsatisfiedLinkError: Native Library /usr/mware/SimpleAPI/voltage-simple-api-java-05.12.0000-Linux-x86_64-64b-r234867/lib/libvibesimplejava.so already loaded in another classloader

This happens because the dylib has already been loaded once by the taskmanger, how can I mitigate this? It seems problematic if two applications are loading the same dylib. 

Thanks,

Vishwas

[Yang Wang]

Hi Vishwas,

I think it just because dylib is loaded more than once in a jvm process(TaskManager).

Multiple tasks are deployed in one TaskManager and running in different threads.

So if you want to make the dylib only loaded once, maybe you use the parent classloader.

You could use the the following config option to set the packages to be loaded by parent classloader.

classloader.parent-first-patterns.additional: xxx.yyy.*

Have a try :)

Best,

Yang

Vishwas Siravara <[hidden email]> 于2019年8月28日周三 下午9:40写道：

Yes this is exactly what happens , as a work around I created a small jar file which has code to load the dylib and I placed it under the lib folder , this library is in provided scope in my actual job, so the dylib gets loaded only once when the tm/jm jvm starts . 

What I found interesting in my old approach was even when I check whether the dylib has already been loaded in the current thread , and if it is I still get the unsatisfied link error even though that dylib is loaded in the task manager . 

On Wed, Aug 28, 2019 at 7:04 AM Aleksey Pak <[hidden email]> wrote:

Hi Vishwas,

There is a known issue in the Flink Jira project [1].

Is it possible that you have encountered the same problem?

[1]: https://issues.apache.org/jira/browse/FLINK-11402

Regards,

Aleksey

On Tue, Aug 27, 2019 at 8:03 AM Vishwas Siravara <[hidden email]> wrote:

Hi Jörn, 
I tried that. Here is my snippet : 

String[] loadedlibs =  getLoadedLibraries(Thread.currentThread().getContextClassLoader());
if(!containsVibeSimpleLib(loadedlibs)) {
    System.loadLibrary("vibesimplejava");
}

Now I get the exception Unexpected errorjava.lang.UnsatisfiedLinkError: com.voltage.securedata.enterprise.ConstantsNative.DIGEST_MD5()I which means that it could not find vibesimplejava in the loaded libs but I know that the if was not executed because vibesimplejava was present in loadedlibs( the control never went inside the if block. Any other suggestions?

Thanks,

Vishwas

On Tue, Aug 27, 2019 at 12:25 AM Jörn Franke <[hidden email]> wrote:

I don’t know Dylibs in detail, but can you call a static method where it checks if it has been already executed and if not then it loads the library (Singleton pattern)?

Am 27.08.2019 um 06:39 schrieb Vishwas Siravara <[hidden email]>:

Hi guys,

I have a flink application that loads a dylib like this 

System.loadLibrary("vibesimplejava");

The application runs fine , when I restart the job I get this exception : 

com.visa.aip.cryptolib.aipcyptoclient.EncryptionException: Unexpected errorjava.lang.UnsatisfiedLinkError: Native Library /usr/mware/SimpleAPI/voltage-simple-api-java-05.12.0000-Linux-x86_64-64b-r234867/lib/libvibesimplejava.so already loaded in another classloader

This happens because the dylib has already been loaded once by the taskmanger, how can I mitigate this? It seems problematic if two applications are loading the same dylib. 

Thanks,

Vishwas