java.lang.NoClassDefFoundError due to Native library issue?

[Hegde, Mahendra]

Hello Team,

 

I am trying to use timezone finding service (https://github.com/dustin-johnson/timezonemap) in my Flink java job. It worked fine in local machine and it worked fine initially in Flink server, but after 2-3 restart of the job it started giving NoClassDefFoundError error-

 

java.lang.NoClassDefFoundError: Could not initialize class com.github.luben.zstd.ZstdInputStream

                at org.apache.commons.compress.compressors.zstandard.ZstdCompressorInputStream.<init>(ZstdCompressorInputStream.java:43)

                at us.dustinj.timezonemap.data.DataLocator.getDataInputStream(DataLocator.java:22)

                at us.dustinj.timezonemap.TimeZoneMap.forRegion(TimeZoneMap.java:92)

                at us.dustinj.timezonemap.TimeZoneMap.forEverywhere(TimeZoneMap.java:60)

                at dtap.service.TimezoneService.<init>(TimezoneService.java:17)

                at functions.processors.KernelIntgrationProcessor.processElement(KernelIntgrationProcessor.java:47)

                at functions.processors.KernelIntgrationProcessor.processElement(KernelIntgrationProcessor.java:23)

                at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)

                at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)

                at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)

                at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)

                at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

                at java.lang.Thread.run(Thread.java:748)

 

 

When I looked at the code of ZstdCompressorInputStream class, it is loading native libraries. And I found from below page that native library unloading issue could be the reason for this.

https://www.ibm.com/support/knowledgecenter/SS7K4U_8.5.5/com.ibm.websphere.zseries.doc/ae/rtrb_classload_viewer.html

Does anyone has faced similar issue or has idea to solve this problem?

 

Thanks,

Mahendra

 

[Ruidong Li]

I've come across a similar issue before, the reason is that for a dynamic link library(so/dll), the there can only be one classloader to load it.

When restart/failover happens in flink, the JVM will not exit but only create a new classloader which leads to multiple loading to the same so/dll, here same so/dll means the full path of a so/dll , unless the previous classloader has been garbage collected.

So here comes the solution

1. load a new so/dll, you can rename the so file, change the path of so, this is how flink rocksdb stateback works.

2. try using a fixed classloader 

Hegde, Mahendra <[hidden email]> 于2019年12月20日周五 下午2:19写道：

Hello Team,

 

I am trying to use timezone finding service (https://github.com/dustin-johnson/timezonemap) in my Flink java job. It worked fine in local machine and it worked fine initially in Flink server, but after 2-3 restart of the job it started giving NoClassDefFoundError error-

 

java.lang.NoClassDefFoundError: Could not initialize class com.github.luben.zstd.ZstdInputStream

                at org.apache.commons.compress.compressors.zstandard.ZstdCompressorInputStream.<init>(ZstdCompressorInputStream.java:43)

                at us.dustinj.timezonemap.data.DataLocator.getDataInputStream(DataLocator.java:22)

                at us.dustinj.timezonemap.TimeZoneMap.forRegion(TimeZoneMap.java:92)

                at us.dustinj.timezonemap.TimeZoneMap.forEverywhere(TimeZoneMap.java:60)

                at dtap.service.TimezoneService.<init>(TimezoneService.java:17)

                at functions.processors.KernelIntgrationProcessor.processElement(KernelIntgrationProcessor.java:47)

                at functions.processors.KernelIntgrationProcessor.processElement(KernelIntgrationProcessor.java:23)

                at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)

                at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)

                at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)

                at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)

                at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

                at java.lang.Thread.run(Thread.java:748)

 

 

When I looked at the code of ZstdCompressorInputStream class, it is loading native libraries. And I found from below page that native library unloading issue could be the reason for this.

https://www.ibm.com/support/knowledgecenter/SS7K4U_8.5.5/com.ibm.websphere.zseries.doc/ae/rtrb_classload_viewer.html

Does anyone has faced similar issue or has idea to solve this problem?

 

Thanks,

Mahendra