FLINK-5236. flink 1.1.3 and modifying the classpath to use another version of fasterxml.

[Estela Gallardo Zapata]

Hello everyone,

I'm new in  the flink mailing-list :)  and I'll try to be respectful and helpful.

I'm writing  to this mailing list, due to an issue l have with Flink 1.1.3 and my application.jar, related to this previous thread: https://issues.apache.org/jira/browse/FLINK-5236

I use Intelli-J to run locally my maven application and it runs correctly with flink 1.1.3 and scala 2.10. 

But when I try to run my application's jar in the cluster, it seems that there is an issue with library faxterxml. 

It seems that Flink uses a different version of this library and, as it overwrites the user-classpath, I can't get my application running in the cluster.

I've tried modifying the Hadoop-conf file, making a flat-jar, modifying HADOOP_USER_CLASSPATH_FIRST, copy my application's jar directly in flink/lib...

Could someone please give me some more clues or help me clarify the explanation in the mailling list?

Thank you very much.

Estela.

[Till Rohrmann]

Hi Estela,

it's correct that Flink's runtime has a dependency on 

com.fasterxml.jackson.core:jackson-core:jar:2.7.4:compile

com.fasterxml.jackson.core:jackson-databind:jar:2.7.4:compile

com.fasterxml.jackson.core:jackson-annotations:jar:2.7.4:compile

First of all it's important to understand where the additional dependency comes from. Does your application has a dependency on fasterxml or does the Flink cluster does not start because Hadoop brings in a different version of fasterxml? In case of the latter, which Hadoop version are you running? In case of the former, can you change the user code fasterxml version to 2.7.4?

The job manager log under log/flink-xxxx-jobmanager-xxxx.log will print all dependencies in the classpath of the job manager. This information would also be interesting.

As a short term solution, you could try to build a custom Flink version where you adapt the jackson.version in the pom.xml so that it's matching your fasterxml version. As a midterm solution, it might be necessary to shade fasterxml in Flink and as a long term solution, we should try to not expose all system classloader dependencies to the user code class loader. But that's something Flink has to do.

Cheers,

Till

On Tue, Jan 10, 2017 at 10:10 PM, Estela Gallardo Zapata <[hidden email]> wrote:

Hello everyone,

I'm new in  the flink mailing-list :)  and I'll try to be respectful and helpful.

I'm writing  to this mailing list, due to an issue l have with Flink 1.1.3 and my application.jar, related to this previous thread: https://issues.apache.org/jira/browse/FLINK-5236

I use Intelli-J to run locally my maven application and it runs correctly with flink 1.1.3 and scala 2.10. 

But when I try to run my application's jar in the cluster, it seems that there is an issue with library faxterxml. 

It seems that Flink uses a different version of this library and, as it overwrites the user-classpath, I can't get my application running in the cluster.

I've tried modifying the Hadoop-conf file, making a flat-jar, modifying HADOOP_USER_CLASSPATH_FIRST, copy my application's jar directly in flink/lib...

Could someone please give me some more clues or help me clarify the explanation in the mailling list?

Thank you very much.

Estela.