Re: Proper way of adding external jars

Posted by Gyula Fóra on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Proper-way-of-adding-external-jars-tp10102p10114.html

Hi Scott, 

Thanks, I am familiar with the ways you suggested. Unfortunately packaging everything together is not really an option in our case, we specifically want to avoid having to do this as many people will set up their own builds and they will inevitable fail to include everything necessary with the correct setup.

Including it in the lib folder would mean I have to copy/remove jars all the time when deploying new jobs if I don't want to have dependency clashes and also it doesn't seem to be a clean solution for something so simple.

I thought the -C option would actually do what I am looking for in an elegant way that's why I am trying to understand what happened.

Cheers,

Gyula

Scott Kidder <[hidden email]> ezt írta (időpont: 2016. nov. 14., H, 18:49):

Hi Gyula,

I've typically added external library dependencies to my own application JAR as shaded-dependencies. This ensures that all dependencies are included with my application while being distributed to Flink Job Manager & Task Manager instances.

Another approach is to place these external JARs in the 'lib' sub-directory of your Flink installation. Keep in mind that the external JARs must be installed on every Flink node where your application is expected to run. This works well for dependencies that are large in size or used by multiple Flink applications in your cluster (avoid duplication of dependencies).

Best,

--Scott Kidder

On Mon, Nov 14, 2016 at 7:59 AM, Gyula Fóra <[hidden email]> wrote:

Hi,

I have been trying to use the -C flag to add external jars with user code and I have observed some strange behaviour.

What I am trying to do is the following:

I have 2 jars, JarWithMain.jar contains the main class and UserJar.jar contains some classes that the main method will eventually execute and also depends on classes from JarWithMain.

Running this works:

flink run .... -C UserJar.jar -c MainMethod JarWithMain.jar args...

Running this leads to no class def found errors in the StreamTask initialization where it reads the functions from the config:

flink run .... -C JarWithMain.jar -c MainMethod UserJar.jar  args...

Did I miss something?

Cheers,

Gyula