Kafka exceptions in Flink log file

[Alexander Smirnov]

I see a lot of messages in flink log like below. What's the cause?

02 Apr 2018 04:09:13,554 ERROR org.apache.kafka.clients.producer.internals.Sender - Uncaught error in kafka producer I/O thread:

org.apache.kafka.common.KafkaException: Error registering mbean kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1

        at org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:163)

        at org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:81)

        at org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:504)

        at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:255)

        at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:240)

        at org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:811)

        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:353)

        at org.apache.kafka.common.network.Selector.poll(Selector.java:326)

        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)

        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)

        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)

        at java.lang.Thread.run(Thread.java:748)

Caused by: javax.management.InstanceAlreadyExistsException: kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1

        at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)

        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)

        at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)

        at org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:161)

        ... 11 more

02 Apr 2018 04:09:13,673 ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in kafka-producer-network-thread | producer-3:

java.lang.NoClassDefFoundError: org/apache/kafka/clients/NetworkClient$1

        at org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:583)

        at org.apache.kafka.clients.NetworkClient.handleDisconnections(NetworkClient.java:705)

        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:443)

        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)

        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.NetworkClient$1

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        ... 6 more

Thank you,

Alex

[Timo Walther]

Hi Alex,

which version of Flink are you running? There were some class loading
issues with Kafka recently. I would try it with the newest Flink
version. Otherwise ClassNotFoundException usually indicates that
something is wrong with your dependencies. Maybe you can share your
pom.xml with us.

Regards,
Timo

Am 02.04.18 um 13:32 schrieb Alexander Smirnov:

> I see a lot of messages in flink log like below. What's the cause?
>
>
> 02 Apr 2018 04:09:13,554 ERROR
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> kafka producer I/O thread:
> org.apache.kafka.common.KafkaException: Error registering mbean
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
>         at
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:163)
>         at
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:81)
>         at
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:504)
>         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:255)
>         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:240)
>         at
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:811)
>         at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:353)
>         at
> org.apache.kafka.common.network.Selector.poll(Selector.java:326)
>         at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.management.InstanceAlreadyExistsException:
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
>         at
> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
>         at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>         at
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:161)
>         ... 11 more
> 02 Apr 2018 04:09:13,673 ERROR
> org.apache.kafka.common.utils.KafkaThread - Uncaught exception in
> kafka-producer-network-thread | producer-3:
> java.lang.NoClassDefFoundError: org/apache/kafka/clients/NetworkClient$1
>         at
> org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:583)
>         at
> org.apache.kafka.clients.NetworkClient.handleDisconnections(NetworkClient.java:705)
>         at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:443)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.kafka.clients.NetworkClient$1
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 6 more
>
>
> Thank you,
> Alex

[Alexander Smirnov]

Hi Timo,

it is the latest released version - 1.4.2

This only happens when a job falls into a restart loop and stays in it for 20 minutes or so.

Looks like for each restart, Flink loads classes anew, but previously loaded classes are not garbage collected for some reason (still referenced?)

Very soon, JVM runs out of Metaspace and this error occurs.

Thank you,

Alex

On Tue, Apr 3, 2018 at 4:24 PM Timo Walther <[hidden email]> wrote:

Hi Alex,

which version of Flink are you running? There were some class loading
issues with Kafka recently. I would try it with the newest Flink
version. Otherwise ClassNotFoundException usually indicates that
something is wrong with your dependencies. Maybe you can share your
pom.xml with us.

Regards,
Timo

Am 02.04.18 um 13:32 schrieb Alexander Smirnov:
> I see a lot of messages in flink log like below. What's the cause?
>
>
> 02 Apr 2018 04:09:13,554 ERROR
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> kafka producer I/O thread:
> org.apache.kafka.common.KafkaException: Error registering mbean
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
>         at
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:163)
>         at
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:81)
>         at
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:504)
>         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:255)
>         at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:240)
>         at
> org.apache.kafka.common.network.Selector$SelectorMetrics.maybeRegisterConnectionMetrics(Selector.java:811)
>         at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:353)
>         at
> org.apache.kafka.common.network.Selector.poll(Selector.java:326)
>         at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:433)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.management.InstanceAlreadyExistsException:
> kafka.producer:type=producer-node-metrics,client-id=producer-1,node-id=node-1
>         at
> com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
>         at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
>         at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
>         at
> org.apache.kafka.common.metrics.JmxReporter.reregister(JmxReporter.java:161)
>         ... 11 more
> 02 Apr 2018 04:09:13,673 ERROR
> org.apache.kafka.common.utils.KafkaThread - Uncaught exception in
> kafka-producer-network-thread | producer-3:
> java.lang.NoClassDefFoundError: org/apache/kafka/clients/NetworkClient$1
>         at
> org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:583)
>         at
> org.apache.kafka.clients.NetworkClient.handleDisconnections(NetworkClient.java:705)
>         at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:443)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224)
>         at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.kafka.clients.NetworkClient$1
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 6 more
>
>
> Thank you,
> Alex