java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer

[syed]

Hi;

I am trying to run simple worcount application using the kafka consumer so
that the data sources remain available and I can trigger checkpoints. When
deploying the application over flink 1.7.2 cluster, I am facing the
*java.lang.NoClassDefFoundError:
org/apache/kafka/common/serialization/ByteArrayDeserializer* error.

The complete trace of error is as follows;java.lang.NoClassDefFoundError:
org/apache/kafka/common/serialization/ByteArrayDeserializer at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.setDeserializer(FlinkKafkaConsumer09.java:299)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:218)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:156)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:140)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:107)at
org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at
java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)at
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)at

org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)at
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)Caused
by: java.lang.ClassNotFoundException:
org.apache.kafka.common.serialization.ByteArrayDeserializerat
java.net.URLClassLoader.findClass(URLClassLoader.java:382)at
java.lang.ClassLoader.loadClass(ClassLoader.java:418)at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)at
java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 20 more

I am using kafka_2.11-2.3.0, and flink 1.7.2. The application code comiples and jar is
created, but the error occurs when running the jar over flink cluster. The
code snippet of word count for kafka sources is as follows;

DataStream text;if (params.has("topic") && params.has("bootstrap.servers")
&¶ms.has("zookeeper.connect") && params.has("group.id")){text =
env.addSource(new FlinkKafkaConsumer08( params.get("topic"),new
SimpleStringSchema(),params.getProperties()).setStartFromEarliest());} else
{System.out.println("Executing WordCount example with default input data
set.");System.out.println("Use --topic  --bootstrap.servers  "
+"--zookeeper.connect  --group.id  specify the topic info.");text =
env.fromElements(WordCountData.WORDS);

Please guide me how to fix this error.
I also tried to use FlinkKafkaConsumer09, but not successful.Thank
you.Looking forward;Syed



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Congxian Qiu]

Hi

I encountered same problem before and found that the `NoClassDefFoundError` is not the root cause, maybe you can check the jm/tm log to see if this is your case.

syed <[hidden email]> 于2019年12月24日周二 下午7:30写道：

Hi; I am trying to run simple worcount application using the kafka consumer so that the data sources remain available and I can trigger checkpoints. When deploying the application over flink 1.7.2 cluster, I am facing the java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer error. The complete trace of error is as follows; java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.setDeserializer(FlinkKafkaConsumer09.java:299) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:218) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:156) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:140) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09.(FlinkKafkaConsumer09.java:107)at org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813) at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050) at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126) at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126) Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArrayDeserializer at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 20 more I am using kafka_2.11-2.3.0, and flink 1.7.2. The application code comiples and jar is created, but the error occurs when running the jar over flink cluster. The code snippet of word count for kafka sources is as follows; DataStream text; if (params.has("topic") && params.has("bootstrap.servers") && params.has("zookeeper.connect") && params.has("group.id")){ text = env.addSource(new FlinkKafkaConsumer08( params.get("topic"), new SimpleStringSchema(), params.getProperties() ).setStartFromEarliest()); } else { System.out.println("Executing WordCount example with default input data set."); System.out.println("Use --topic --bootstrap.servers " +"--zookeeper.connect --group.id specify the topic info."); text = env.fromElements(WordCountData.WORDS); Please guide me how to fix this error. I also tried to use FlinkKafkaConsumer09, but not successful. Thank you. Looking forward; Syed

---

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[jingjing bai]

Typically , NoClassDefFoundError  is caused   by jar conflicts ,  it means, there are two same class and not same version jar in your classpath,  

I suggest you can check which jar  is this class from ,and then to confirm is there more too than one jar in u classpath.  if is and remove unuse jar .

I hope this can help you.

best