(DEPRECATED) Apache Flink User Mailing List archive.

Akka version conflict running on Flink cluster

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
Wouter Zorgdrager
Reply | Threaded
Open this post in threaded view
♦
♦
|

Akka version conflict running on Flink cluster

Wouter Zorgdrager
Hi,

I think I'm running into an Akka version conflict when running a Flink job on a cluster.

The current situation:
- Flink cluster on Flink 1.4.2 (using Docker)
- Flink job which uses twitter4s [1] library and Akka version 2.5.8

In my Flink job I try to 'shutdown' an Akka actor from the twitter4s library.
This results in a whole taskmanager crashing with the following stacktrace:

taskrunner_1  | 2018-06-11 09:03:14,454 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source -> Sink: Unnamed (0ba7f7f259eee06fe2f7d783c868179b)
taskrunner_1  | Uncaught error from thread [twitter4s-streaming-akka.actor.default-dispatcher-288]: loader constraint violation: when resolving method "akka.actor.ActorCell$$anonfun$3.<init>(Lakka/actor/ActorCell;)V" the class loader (instance of org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, akka/actor/ActorCell$$anonfun$3, have different Class objects for the type akka/actor/ActorCell used in the signature, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[twitter4s-streaming]
taskrunner_1  | java.lang.LinkageError: loader constraint violation: when resolving method "akka.actor.ActorCell$$anonfun$3.<init>(Lakka/actor/ActorCell;)V" the class loader (instance of org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, akka/actor/ActorCell$$anonfun$3, have different Class objects for the type akka/actor/ActorCell used in the signature
taskrunner_1  |     at akka.actor.ActorCell.invoke(ActorCell.scala:499)
taskrunner_1  |     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
taskrunner_1  |     at akka.dispatch.Mailbox.run(Mailbox.scala:224)
taskrunner_1  |     at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
taskrunner_1  | 2018-06-11 09:03:14,984 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
taskrunner_1  | 2018-06-11 09:03:14,985 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
taskrunner_1  | Exception in thread "twitter4s-streaming-shutdown-hook-1" java.lang.NoClassDefFoundError: akka/actor/CoordinatedShutdown$$anonfun$totalTimeout$1
taskrunner_1  |     at akka.actor.CoordinatedShutdown.totalTimeout(CoordinatedShutdown.scala:515)
taskrunner_1  |     at akka.actor.CoordinatedShutdown$$anonfun$initJvmHook$1.apply(CoordinatedShutdown.scala:217)
taskrunner_1  |     at akka.actor.CoordinatedShutdown$$anon$2.run(CoordinatedShutdown.scala:547)
taskrunner_1  | Caused by: java.lang.ClassNotFoundException: akka.actor.CoordinatedShutdown$$anonfun$totalTimeout$1
taskrunner_1  |     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
taskrunner_1  |     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
taskrunner_1  |     at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
taskrunner_1  |     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
taskrunner_1  |     ... 3 more

To me, it looks like an version conflict. Any suggestions how to solve this?

Thanks!
Wouter
Piotr Nowojski
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Akka version conflict running on Flink cluster

Piotr Nowojski
Hi,

Please take a look on this thread first:

Piotrek

On 11 Jun 2018, at 11:16, Wouter Zorgdrager <[hidden email]> wrote:

Hi,

I think I'm running into an Akka version conflict when running a Flink job on a cluster.

The current situation:
- Flink cluster on Flink 1.4.2 (using Docker)
- Flink job which uses twitter4s [1] library and Akka version 2.5.8

In my Flink job I try to 'shutdown' an Akka actor from the twitter4s library.
This results in a whole taskmanager crashing with the following stacktrace:

taskrunner_1  | 2018-06-11 09:03:14,454 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source -> Sink: Unnamed (0ba7f7f259eee06fe2f7d783c868179b)
taskrunner_1  | Uncaught error from thread [twitter4s-streaming-akka.actor.default-dispatcher-288]: loader constraint violation: when resolving method "akka.actor.ActorCell$$anonfun$3.<init>(Lakka/actor/ActorCell;)V" the class loader (instance of org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, akka/actor/ActorCell$$anonfun$3, have different Class objects for the type akka/actor/ActorCell used in the signature, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[twitter4s-streaming]
taskrunner_1  | java.lang.LinkageError: loader constraint violation: when resolving method "akka.actor.ActorCell$$anonfun$3.<init>(Lakka/actor/ActorCell;)V" the class loader (instance of org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoaders$ChildFirstClassLoader) of the current class, akka/actor/ActorCell, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, akka/actor/ActorCell$$anonfun$3, have different Class objects for the type akka/actor/ActorCell used in the signature
taskrunner_1  |     at akka.actor.ActorCell.invoke(ActorCell.scala:499)
taskrunner_1  |     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
taskrunner_1  |     at akka.dispatch.Mailbox.run(Mailbox.scala:224)
taskrunner_1  |     at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
taskrunner_1  |     at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
taskrunner_1  | 2018-06-11 09:03:14,984 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
taskrunner_1  | 2018-06-11 09:03:14,985 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
taskrunner_1  | Exception in thread "twitter4s-streaming-shutdown-hook-1" java.lang.NoClassDefFoundError: akka/actor/CoordinatedShutdown$$anonfun$totalTimeout$1
taskrunner_1  |     at akka.actor.CoordinatedShutdown.totalTimeout(CoordinatedShutdown.scala:515)
taskrunner_1  |     at akka.actor.CoordinatedShutdown$$anonfun$initJvmHook$1.apply(CoordinatedShutdown.scala:217)
taskrunner_1  |     at akka.actor.CoordinatedShutdown$$anon$2.run(CoordinatedShutdown.scala:547)
taskrunner_1  | Caused by: java.lang.ClassNotFoundException: akka.actor.CoordinatedShutdown$$anonfun$totalTimeout$1
taskrunner_1  |     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
taskrunner_1  |     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
taskrunner_1  |     at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:128)
taskrunner_1  |     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
taskrunner_1  |     ... 3 more

To me, it looks like an version conflict. Any suggestions how to solve this?

Thanks!
Wouter
Free forum by Nabble Edit this page