Flink grpc-netty-shaded NoClassDefFoundError

[dhanesh arole]

Hello all, 

We are running on Flink 1.6.2 and have a couple of streaming jobs running on it. We are intermittently seeing  Java.lang.NoClassDefFoundError with below stack trace[1]. The Flink job was working fine but due to recent this errors, task managers are now frequently crashing and causing to restart the job. We haven't changed any dep version / Flink version, so we are not sure if it's related to version mismatch. This is only happening with classes from io.grpc.netty.shaded package. Even the classes that FlinkClassLoader is not able to find are actually present in fat jar built using shadowJar - (validated by looking at jar tf flink-job.jar).

Has anyone faced such issue before? Any pointers on how to debug this would be really helpful.

I am attaching output of './gradlew dep' as well for reference, so as to validate if we don't have some wrong or flaky dependencies.

[1] Stack trace: Task manager Java.lang.NoClassDefFoundError

"java.lang.NoClassDefFoundError: io/grpc/netty/shaded/io/netty/channel/AbstractChannel$AbstractUnsafe$8
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.deregister(AbstractChannel.java:817)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.fireChannelInactiveAndDeregister(AbstractChannel.java:777)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:760)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$8
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 13 more
"

[dhanesh arole]

Just to give you more context, 

We are using `com.google.cloud.bigtable` as well in our job dep. Could it be due to shaded plugin issue with `bigtable-hbase-2.x` ?

On Tue, Oct 22, 2019 at 2:06 PM dhanesh arole <[hidden email]> wrote:

Hello all, 

We are running on Flink 1.6.2 and have a couple of streaming jobs running on it. We are intermittently seeing  Java.lang.NoClassDefFoundError with below stack trace[1]. The Flink job was working fine but due to recent this errors, task managers are now frequently crashing and causing to restart the job. We haven't changed any dep version / Flink version, so we are not sure if it's related to version mismatch. This is only happening with classes from io.grpc.netty.shaded package. Even the classes that FlinkClassLoader is not able to find are actually present in fat jar built using shadowJar - (validated by looking at jar tf flink-job.jar).

Has anyone faced such issue before? Any pointers on how to debug this would be really helpful.

I am attaching output of './gradlew dep' as well for reference, so as to validate if we don't have some wrong or flaky dependencies.

[1] Stack trace: Task manager Java.lang.NoClassDefFoundError

"java.lang.NoClassDefFoundError: io/grpc/netty/shaded/io/netty/channel/AbstractChannel$AbstractUnsafe$8
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.deregister(AbstractChannel.java:817)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.fireChannelInactiveAndDeregister(AbstractChannel.java:777)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:760)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$8
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 13 more
"

[Piotr Nowojski-3]

Hi,

Have you checked build of your job for a dependency convergence errors? Either automatically or manually (`mvn dependency:tree` command)? Look for the version clashes for the dependencies that are pulling in `io/grpc/netty/shaded/io/netty/channel/AbstractChannel` class (grpc-netty?).

It could be an error purely with dependencies of your job. Alternatively if you are using `flink-connector-gcp-pubsub` connector, it can be a clash between something that you are pulling in and this connector (I think this is the only thing from Flink that depends on grpc-netty). 

Either way, if it’s a dependency conflict, you have to fix it somehow. As long as your application depends on different versions of the same dependency you might always keep getting class/method not found errors. Potential solutions:

- drop the dependency

- converge the versions of this dependency (enforce your build system to always use the same version of it in all of the places)

- shade it

Piotrek

On 22 Oct 2019, at 10:40, dhanesh arole <[hidden email]> wrote:

Just to give you more context, 

We are using `com.google.cloud.bigtable` as well in our job dep. Could it be due to shaded plugin issue with `bigtable-hbase-2.x` ?

On Tue, Oct 22, 2019 at 2:06 PM dhanesh arole <[hidden email]> wrote:

Hello all, 

We are running on Flink 1.6.2 and have a couple of streaming jobs running on it. We are intermittently seeing  Java.lang.NoClassDefFoundError with below stack trace[1]. The Flink job was working fine but due to recent this errors, task managers are now frequently crashing and causing to restart the job. We haven't changed any dep version / Flink version, so we are not sure if it's related to version mismatch. This is only happening with classes from io.grpc.netty.shaded package. Even the classes that FlinkClassLoader is not able to find are actually present in fat jar built using shadowJar - (validated by looking at jar tf flink-job.jar).

Has anyone faced such issue before? Any pointers on how to debug this would be really helpful.

I am attaching output of './gradlew dep' as well for reference, so as to validate if we don't have some wrong or flaky dependencies.

[1] Stack trace: Task manager Java.lang.NoClassDefFoundError

"java.lang.NoClassDefFoundError: io/grpc/netty/shaded/io/netty/channel/AbstractChannel$AbstractUnsafe$8
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.deregister(AbstractChannel.java:817)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.fireChannelInactiveAndDeregister(AbstractChannel.java:777)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:760)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$8
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 13 more
"