Elasticsink sometimes gives NoClassDefFoundError

[Jayant Ameta]

Hi,

I see the following error sometimes on my flink job, even though the class is present in my uber jar.

java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:111) ... 17 common frames omitted Wrapped by: org.elasticsearch.ElasticsearchException: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.elasticsearch.transport.netty3.Netty3Transport.exceptionCaught(Netty3Transport.java:325) at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.exceptionCaught(Netty3MessageChannelHandler.java:83) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ... 23 frames truncated 

[Chesnay Schepler]

Could you clarify under what circumstances you see this issue? You say "sometimes"; is the job running normally and then failing due to this error? Does it happen when submitting/canceling/restarting a job etc.

On 08/01/2020 12:20, Jayant Ameta wrote:

Hi,

I see the following error sometimes on my flink job, even though the class is present in my uber jar.

java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:111) ... 17 common frames omitted Wrapped by: org.elasticsearch.ElasticsearchException: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.elasticsearch.transport.netty3.Netty3Transport.exceptionCaught(Netty3Transport.java:325) at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.exceptionCaught(Netty3MessageChannelHandler.java:83) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ... 23 frames truncated 

[Arvid Heise-3]

Hi Jayant,

if you only see it sometimes that indicates that you have it in two different versions of the connectors where class loader order is non-deterministic. Could you post the classpath?

Btw, it's always good to add which Flink version you use.

Best,

Arvid

On Wed, Jan 8, 2020 at 12:20 PM Jayant Ameta <[hidden email]> wrote:

Hi,

I see the following error sometimes on my flink job, even though the class is present in my uber jar.

java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:111) ... 17 common frames omitted Wrapped by: org.elasticsearch.ElasticsearchException: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.elasticsearch.transport.netty3.Netty3Transport.exceptionCaught(Netty3Transport.java:325) at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.exceptionCaught(Netty3MessageChannelHandler.java:83) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ... 23 frames truncated 

Jayant 

[Jayant Ameta]

Hi,

The elastic connector is packaged in the uber jar that is submitted. There is only 1 version of the connector: flink-connector-elasticsearch5_2.11:1.7.1

I'm using Flink 1.7.1

I couldn't figure out whether this error causes the job to fail, or whether I see this error when the job is restarting after some other failure.

But, the occurrence of this error and job restarts is correlated.

On Wed, Jan 8, 2020 at 6:47 PM Arvid Heise <[hidden email]> wrote:

Hi Jayant,

if you only see it sometimes that indicates that you have it in two different versions of the connectors where class loader order is non-deterministic. Could you post the classpath?

Btw, it's always good to add which Flink version you use.

Best,

Arvid

On Wed, Jan 8, 2020 at 12:20 PM Jayant Ameta <[hidden email]> wrote:

Hi,

I see the following error sometimes on my flink job, even though the class is present in my uber jar.

java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:111) ... 17 common frames omitted Wrapped by: org.elasticsearch.ElasticsearchException: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.elasticsearch.transport.netty3.Netty3Transport.exceptionCaught(Netty3Transport.java:325) at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.exceptionCaught(Netty3MessageChannelHandler.java:83) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ... 23 frames truncated 

Jayant 

[Jayant Ameta]

Also, the ES version I'm using is 5.6.7

On Thu, Jan 9, 2020 at 10:39 AM Jayant Ameta <[hidden email]> wrote:

Hi,

The elastic connector is packaged in the uber jar that is submitted. There is only 1 version of the connector: flink-connector-elasticsearch5_2.11:1.7.1

I'm using Flink 1.7.1

I couldn't figure out whether this error causes the job to fail, or whether I see this error when the job is restarting after some other failure.

But, the occurrence of this error and job restarts is correlated.

Jayant Ameta

On Wed, Jan 8, 2020 at 6:47 PM Arvid Heise <[hidden email]> wrote:

Hi Jayant,

if you only see it sometimes that indicates that you have it in two different versions of the connectors where class loader order is non-deterministic. Could you post the classpath?

Btw, it's always good to add which Flink version you use.

Best,

Arvid

On Wed, Jan 8, 2020 at 12:20 PM Jayant Ameta <[hidden email]> wrote:

Hi,

I see the following error sometimes on my flink job, even though the class is present in my uber jar.

java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:111) ... 17 common frames omitted Wrapped by: org.elasticsearch.ElasticsearchException: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.elasticsearch.transport.netty3.Netty3Transport.exceptionCaught(Netty3Transport.java:325) at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.exceptionCaught(Netty3MessageChannelHandler.java:83) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ... 23 frames truncated 

Jayant 

[Arvid Heise-3]

I don't see a particular reason why you see this behavior. Chesnay's explanation is the only plausible way that this behavior can happen.

I fear that without a specific log, we cannot help further.

On Fri, Jan 10, 2020 at 5:06 AM Jayant Ameta <[hidden email]> wrote:

Also, the ES version I'm using is 5.6.7

Jayant

On Thu, Jan 9, 2020 at 10:39 AM Jayant Ameta <[hidden email]> wrote:

Hi,

The elastic connector is packaged in the uber jar that is submitted. There is only 1 version of the connector: flink-connector-elasticsearch5_2.11:1.7.1

I'm using Flink 1.7.1

I couldn't figure out whether this error causes the job to fail, or whether I see this error when the job is restarting after some other failure.

But, the occurrence of this error and job restarts is correlated.

Jayant Ameta

On Wed, Jan 8, 2020 at 6:47 PM Arvid Heise <[hidden email]> wrote:

Hi Jayant,

if you only see it sometimes that indicates that you have it in two different versions of the connectors where class loader order is non-deterministic. Could you post the classpath?

Btw, it's always good to add which Flink version you use.

Best,

Arvid

On Wed, Jan 8, 2020 at 12:20 PM Jayant Ameta <[hidden email]> wrote:

Hi,

I see the following error sometimes on my flink job, even though the class is present in my uber jar.

java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:111) ... 17 common frames omitted Wrapped by: org.elasticsearch.ElasticsearchException: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.elasticsearch.transport.netty3.Netty3Transport.exceptionCaught(Netty3Transport.java:325) at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.exceptionCaught(Netty3MessageChannelHandler.java:83) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ... 23 frames truncated 

Jayant