TaskManager docker image for Beam WordCount failing with ClassNotFound Exception

[Avijit Saha]

Hi,
I am trying the run the Beam WordCount example on Flink runner using docker container-s for 'Jobcluster' and 'TaskManager'.

When I put the Beam Wordcount custom jar in the /opt/flink/usrlib/ dir - the 'taskmanager' docker image  fails at runtime with ClassNotFound Exception for the following:
Caused by: java.lang.ClassNotFoundException: org.apache.beam.runners.core.metrics.MetricUpdates$MetricUpdate:
taskmanager_1  | Caused by: java.lang.ClassNotFoundException: org.apache.beam.runners.core.metrics.MetricUpdates$MetricUpdate
taskmanager_1  |        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
taskmanager_1  |        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
taskmanager_1  |        at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:69)
taskmanager_1  |        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
taskmanager_1  |        ... 68 more

If I  instead put the Beam WordCount jar in the "/opt/flink-1.10.1/lib" dir as follows,
$ ls
flink-dist_2.12-1.10.1.jar        
flink-table-blink_2.12-1.10.1.jar  
flink-table_2.12-1.10.1.jar        
log4j-1.2.17.jar                  
 slf4j-log4j12-1.7.15.jar          
 word-count-beam-bundled-0.1.jar

It runs without any Exception!

Is this the expected behavior? Do we need to always bundle the job-jar in the same lib location as other flink jars?

[Tzu-Li (Gordon) Tai]

Hi,

Assuming that the job jar bundles all the required dependencies (including
the Beam dependencies), making them available under `/opt/flink/usrlib/` in
the container either by mounting or directly adding the job artifacts should
work. AFAIK It is also the recommended way, as opposed to adding them under
`/lib`.

From the exception, what seems to be missing is the Beam dependencies.
Just to get rid of the obvious first: are you sure that the job jar has
those bundled?

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/