Re: Proper way of adding external jars

Posted by Scott Kidder on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Proper-way-of-adding-external-jars-tp10102p10106.html

Hi Gyula,

I've typically added external library dependencies to my own application JAR as shaded-dependencies. This ensures that all dependencies are included with my application while being distributed to Flink Job Manager & Task Manager instances.

Another approach is to place these external JARs in the 'lib' sub-directory of your Flink installation. Keep in mind that the external JARs must be installed on every Flink node where your application is expected to run. This works well for dependencies that are large in size or used by multiple Flink applications in your cluster (avoid duplication of dependencies).

Best,

--Scott Kidder

On Mon, Nov 14, 2016 at 7:59 AM, Gyula Fóra <[hidden email]> wrote:

Hi,

I have been trying to use the -C flag to add external jars with user code and I have observed some strange behaviour.

What I am trying to do is the following:

I have 2 jars, JarWithMain.jar contains the main class and UserJar.jar contains some classes that the main method will eventually execute and also depends on classes from JarWithMain.

Running this works:

flink run .... -C UserJar.jar -c MainMethod JarWithMain.jar args...

Running this leads to no class def found errors in the StreamTask initialization where it reads the functions from the config:

flink run .... -C JarWithMain.jar -c MainMethod UserJar.jar  args...

Did I miss something?

Cheers,

Gyula