Is there a better way to debug ClassNotFoundException from FlinkUserCodeClassLoaders?

[Hao Sun]

Hi, 

I am wondering if there are any protips to figure out what class is not found?

===== Logs ====

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate chained outputs.

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:133)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at java.util.ArrayList.readObject(ArrayList.java:797)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498)

at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459)

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321)

... 5 more

[qi luo]

Hi Hao,

Since Flink is using Child-First class loader, you may try search for the class "com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45” in your fat JAR. Is that an inner class?

Best,

Qi

On Jan 3, 2019, at 7:01 AM, Hao Sun <[hidden email]> wrote:

Hi, 

I am wondering if there are any protips to figure out what class is not found?

===== Logs ====

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate chained outputs.

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:133)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at java.util.ArrayList.readObject(ArrayList.java:797)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498)

at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459)

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321)

... 5 more

[Hao Sun]

Yep, javap shows the class is there, but FlinkUserCodeClassLoaders somehow could not find it suddenly

javap -cp /opt/flink/lib/zendesk-fps-core-assembly-0.1.0.jar 'com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45'

Compiled from "ConnectedStreams.scala"

public final class com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45 extends org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> {

public com.zendesk.fraudprevention.datatypes.MaxwellEvent createInstance(java.lang.Object[]);

public org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> createSerializerInstance(java.lang.Class<com.zendesk.fraudprevention.datatypes.MaxwellEvent>, org.apache.flink.api.common.typeutils.TypeSerializer<?>[]);

public org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase createSerializerInstance(java.lang.Class, org.apache.flink.api.common.typeutils.TypeSerializer[]);

public java.lang.Object createInstance(java.lang.Object[]);

public com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45(com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90, org.apache.flink.api.common.typeutils.TypeSerializer[]);

}

On Wed, Jan 2, 2019 at 6:04 PM qi luo <[hidden email]> wrote:

Hi Hao,

Since Flink is using Child-First class loader, you may try search for the class "com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45” in your fat JAR. Is that an inner class?

Best,

Qi

On Jan 3, 2019, at 7:01 AM, Hao Sun <[hidden email]> wrote:

Hi, 

I am wondering if there are any protips to figure out what class is not found?

===== Logs ====

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate chained outputs.

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(http://OperatorChain.java:133)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at java.util.ArrayList.readObject(ArrayList.java:797)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498)

at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459)

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321)

... 5 more

[Timo Walther]

Hi Hao,

which Flink version are you using? What do you mean with "suddenly", did it work before?

Regards,

Timo

Am 03.01.19 um 07:13 schrieb Hao Sun:

Yep, javap shows the class is there, but FlinkUserCodeClassLoaders somehow could not find it suddenly

javap -cp /opt/flink/lib/zendesk-fps-core-assembly-0.1.0.jar 'com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45'

Compiled from "ConnectedStreams.scala"

public final class com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45 extends org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> {

public com.zendesk.fraudprevention.datatypes.MaxwellEvent createInstance(java.lang.Object[]);

public org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> createSerializerInstance(java.lang.Class<com.zendesk.fraudprevention.datatypes.MaxwellEvent>, org.apache.flink.api.common.typeutils.TypeSerializer<?>[]);

public org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase createSerializerInstance(java.lang.Class, org.apache.flink.api.common.typeutils.TypeSerializer[]);

public java.lang.Object createInstance(java.lang.Object[]);

public com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45(com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90, org.apache.flink.api.common.typeutils.TypeSerializer[]);

}

On Wed, Jan 2, 2019 at 6:04 PM qi luo <[hidden email]> wrote:

Hi Hao,

Since Flink is using Child-First class loader, you may try search for the class "com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45” in your fat JAR. Is that an inner class?

Best,

Qi

On Jan 3, 2019, at 7:01 AM, Hao Sun <[hidden email]> wrote:

Hi, 

I am wondering if there are any protips to figure out what class is not found?

===== Logs ====

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate chained outputs.

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(http://OperatorChain.java:133)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at java.util.ArrayList.readObject(ArrayList.java:797)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498)

at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459)

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321)

... 5 more

[Hao Sun]

I am on Flink 1.7.1 and K8S.

I said "suddenly" because my program worked fine until I added a new MapFunction.

I do not know the details, but I think I know what is causing it

=== Start of Program ===

val stream: DataStream[MaxwellEvent] = <get the stream>

steam.map(new ProblemFunction()) will cause the issue

class ProblemFunction(stringParam: String)(implicit datadog: DatadogClient) extends MapFunction[MaxwellEvent, MaxwellEvent]

=== End of Program ===

I think the class taking curry params caused the issue, after I give up on the curry format, the error disappeared.

I am using https://github.com/sbt/sbt-assembly to assemble the fat jar.

There might be some issue, or config issue with that as well.

I am reading this article, it is a good start for me as well

https://heapanalytics.com/blog/engineering/missing-scala-class-noclassdeffounderror

On Thu, Jan 3, 2019 at 1:08 AM Timo Walther <[hidden email]> wrote:

Hi Hao,

which Flink version are you using? What do you mean with "suddenly", did it work before?

Regards,

Timo

Am 03.01.19 um 07:13 schrieb Hao Sun:

Yep, javap shows the class is there, but FlinkUserCodeClassLoaders somehow could not find it suddenly

javap -cp /opt/flink/lib/zendesk-fps-core-assembly-0.1.0.jar 'com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45'

Compiled from "ConnectedStreams.scala"

public final class com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45 extends org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> {

public com.zendesk.fraudprevention.datatypes.MaxwellEvent createInstance(java.lang.Object[]);

public org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> createSerializerInstance(java.lang.Class<com.zendesk.fraudprevention.datatypes.MaxwellEvent>, org.apache.flink.api.common.typeutils.TypeSerializer<?>[]);

public org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase createSerializerInstance(java.lang.Class, org.apache.flink.api.common.typeutils.TypeSerializer[]);

public java.lang.Object createInstance(java.lang.Object[]);

public com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45(com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90, org.apache.flink.api.common.typeutils.TypeSerializer[]);

}

On Wed, Jan 2, 2019 at 6:04 PM qi luo <[hidden email]> wrote:

Hi Hao,

Since Flink is using Child-First class loader, you may try search for the class "com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45” in your fat JAR. Is that an inner class?

Best,

Qi

On Jan 3, 2019, at 7:01 AM, Hao Sun <[hidden email]> wrote:

Hi, 

I am wondering if there are any protips to figure out what class is not found?

===== Logs ====

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate chained outputs.

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(http://OperatorChain.java:133)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at java.util.ArrayList.readObject(ArrayList.java:797)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498)

at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459)

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321)

... 5 more

[Congxian Qiu]

Hi, Hao Sun

For debugging the `ClassNotFoundException`, maybe the Arthas[1] tool can help.

[1] Arthas: https://github.com/alibaba/arthas

Hao Sun <[hidden email]> 于2019年1月3日周四 下午10:08写道：

I am on Flink 1.7.1 and K8S.

I said "suddenly" because my program worked fine until I added a new MapFunction.

I do not know the details, but I think I know what is causing it

=== Start of Program ===

val stream: DataStream[MaxwellEvent] = <get the stream>

steam.map(new ProblemFunction()) will cause the issue

class ProblemFunction(stringParam: String)(implicit datadog: DatadogClient) extends MapFunction[MaxwellEvent, MaxwellEvent]

=== End of Program ===

I think the class taking curry params caused the issue, after I give up on the curry format, the error disappeared.

I am using https://github.com/sbt/sbt-assembly to assemble the fat jar.

There might be some issue, or config issue with that as well.

I am reading this article, it is a good start for me as well

https://heapanalytics.com/blog/engineering/missing-scala-class-noclassdeffounderror

On Thu, Jan 3, 2019 at 1:08 AM Timo Walther <[hidden email]> wrote:

Hi Hao,

which Flink version are you using? What do you mean with "suddenly", did it work before?

Regards,

Timo

Am 03.01.19 um 07:13 schrieb Hao Sun:

Yep, javap shows the class is there, but FlinkUserCodeClassLoaders somehow could not find it suddenly

javap -cp /opt/flink/lib/zendesk-fps-core-assembly-0.1.0.jar 'com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45'

Compiled from "ConnectedStreams.scala"

public final class com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45 extends org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> {

public com.zendesk.fraudprevention.datatypes.MaxwellEvent createInstance(java.lang.Object[]);

public org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> createSerializerInstance(java.lang.Class<com.zendesk.fraudprevention.datatypes.MaxwellEvent>, org.apache.flink.api.common.typeutils.TypeSerializer<?>[]);

public org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase createSerializerInstance(java.lang.Class, org.apache.flink.api.common.typeutils.TypeSerializer[]);

public java.lang.Object createInstance(java.lang.Object[]);

public com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45(com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90, org.apache.flink.api.common.typeutils.TypeSerializer[]);

}

On Wed, Jan 2, 2019 at 6:04 PM qi luo <[hidden email]> wrote:

Hi Hao,

Since Flink is using Child-First class loader, you may try search for the class "com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45” in your fat JAR. Is that an inner class?

Best,

Qi

On Jan 3, 2019, at 7:01 AM, Hao Sun <[hidden email]> wrote:

Hi, 

I am wondering if there are any protips to figure out what class is not found?

===== Logs ====

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate chained outputs.

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(http://OperatorChain.java:133)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at java.util.ArrayList.readObject(ArrayList.java:797)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498)

at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459)

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321)

... 5 more

--

[Hao Sun]

Thanks Congxian for the tip. Arthas looks great

On Fri, Jan 4, 2019 at 5:42 PM Congxian Qiu <[hidden email]> wrote:

Hi, Hao Sun

For debugging the `ClassNotFoundException`, maybe the Arthas[1] tool can help.

[1] Arthas: https://github.com/alibaba/arthas

Hao Sun <[hidden email]> 于2019年1月3日周四 下午10:08写道：

I am on Flink 1.7.1 and K8S.

I said "suddenly" because my program worked fine until I added a new MapFunction.

I do not know the details, but I think I know what is causing it

=== Start of Program ===

val stream: DataStream[MaxwellEvent] = <get the stream>

steam.map(new ProblemFunction()) will cause the issue

class ProblemFunction(stringParam: String)(implicit datadog: DatadogClient) extends MapFunction[MaxwellEvent, MaxwellEvent]

=== End of Program ===

I think the class taking curry params caused the issue, after I give up on the curry format, the error disappeared.

I am using https://github.com/sbt/sbt-assembly to assemble the fat jar.

There might be some issue, or config issue with that as well.

I am reading this article, it is a good start for me as well

https://heapanalytics.com/blog/engineering/missing-scala-class-noclassdeffounderror

On Thu, Jan 3, 2019 at 1:08 AM Timo Walther <[hidden email]> wrote:

Hi Hao,

which Flink version are you using? What do you mean with "suddenly", did it work before?

Regards,

Timo

Am 03.01.19 um 07:13 schrieb Hao Sun:

Yep, javap shows the class is there, but FlinkUserCodeClassLoaders somehow could not find it suddenly

javap -cp /opt/flink/lib/zendesk-fps-core-assembly-0.1.0.jar 'com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45'

Compiled from "ConnectedStreams.scala"

public final class com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45 extends org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> {

public com.zendesk.fraudprevention.datatypes.MaxwellEvent createInstance(java.lang.Object[]);

public org.apache.flink.api.scala.typeutils.CaseClassSerializer<com.zendesk.fraudprevention.datatypes.MaxwellEvent> createSerializerInstance(java.lang.Class<com.zendesk.fraudprevention.datatypes.MaxwellEvent>, org.apache.flink.api.common.typeutils.TypeSerializer<?>[]);

public org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase createSerializerInstance(java.lang.Class, org.apache.flink.api.common.typeutils.TypeSerializer[]);

public java.lang.Object createInstance(java.lang.Object[]);

public com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45(com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90, org.apache.flink.api.common.typeutils.TypeSerializer[]);

}

On Wed, Jan 2, 2019 at 6:04 PM qi luo <[hidden email]> wrote:

Hi Hao,

Since Flink is using Child-First class loader, you may try search for the class "com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45” in your fat JAR. Is that an inner class?

Best,

Qi

On Jan 3, 2019, at 7:01 AM, Hao Sun <[hidden email]> wrote:

Hi, 

I am wondering if there are any protips to figure out what class is not found?

===== Logs ====

org.apache.flink.streaming.runtime.tasks.StreamTaskException: Could not instantiate chained outputs.

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:324)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:292)

at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(http://OperatorChain.java:133)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: com.zendesk.fraudprevention.examples.ConnectedStreams$$anon$90$$anon$45

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:77)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at java.util.ArrayList.readObject(ArrayList.java:797)

at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2176)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:510)

at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:498)

at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:459)

at org.apache.flink.streaming.api.graph.StreamConfig.getChainedOutputs(StreamConfig.java:321)

... 5 more

--