Guava version conflict

[Flavio Pompermaier]

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Flavio Pompermaier]

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Flavio Pompermaier]

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Flavio Pompermaier]

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Flavio Pompermaier]

Hi Gordon,

any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

Hi Flavio,

I was just doing some end-to-end rebuild Flink + cluster execution with ES sink tests, and it seems like the Guava shading problem isn’t there anymore in the flink-dist jar.

On the `release-1.3` branch, built with Maven 3.0.5, the Guava dependencies in flink-dist are all properly shaded.

Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all worked fine without the Guava conflict issue.

So I’m pretty sure that if the problem still exists for you, the conflict would have came from other dependencies in your code.

Cheers,

Gordon

On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi Gordon,

any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Flavio Pompermaier]

I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1. Didn't you say that you also had non-shaded guava dependencies in the flink dist jar some days ago?

On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

I was just doing some end-to-end rebuild Flink + cluster execution with ES sink tests, and it seems like the Guava shading problem isn’t there anymore in the flink-dist jar.

On the `release-1.3` branch, built with Maven 3.0.5, the Guava dependencies in flink-dist are all properly shaded.

Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all worked fine without the Guava conflict issue.

So I’m pretty sure that if the problem still exists for you, the conflict would have came from other dependencies in your code.

Cheers,

Gordon

On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi Gordon,

any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

That’s actually what I’m trying to figure out right now, what had changed since then ...

On 16 June 2017 at 12:23:57 PM, Flavio Pompermaier ([hidden email]) wrote:

I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1. Didn't you say that you also had non-shaded guava dependencies in the flink dist jar some days ago?

On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

I was just doing some end-to-end rebuild Flink + cluster execution with ES sink tests, and it seems like the Guava shading problem isn’t there anymore in the flink-dist jar.

On the `release-1.3` branch, built with Maven 3.0.5, the Guava dependencies in flink-dist are all properly shaded.

Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all worked fine without the Guava conflict issue.

So I’m pretty sure that if the problem still exists for you, the conflict would have came from other dependencies in your code.

Cheers,

Gordon

On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi Gordon,

any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

Hi Flavio,

It’s most likely related to a problem with Maven.

I’m pretty sure this actually isn’t a problem anymore. Could you verify by rebuilding Flink and see if the problem remains? Thanks a lot.

Best,

Gordon

On 16 June 2017 at 6:25:10 PM, Tzu-Li (Gordon) Tai ([hidden email]) wrote:

That’s actually what I’m trying to figure out right now, what had changed since then ...

On 16 June 2017 at 12:23:57 PM, Flavio Pompermaier ([hidden email]) wrote:

I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1. Didn't you say that you also had non-shaded guava dependencies in the flink dist jar some days ago?

On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

I was just doing some end-to-end rebuild Flink + cluster execution with ES sink tests, and it seems like the Guava shading problem isn’t there anymore in the flink-dist jar.

On the `release-1.3` branch, built with Maven 3.0.5, the Guava dependencies in flink-dist are all properly shaded.

Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all worked fine without the Guava conflict issue.

So I’m pretty sure that if the problem still exists for you, the conflict would have came from other dependencies in your code.

Cheers,

Gordon

On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi Gordon,

any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Flavio Pompermaier]

Ok, I'll let you know as soon as I recompile Flink 1.3.x.

Thanks,

Flavio

On Mon, Jun 19, 2017 at 7:26 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

It’s most likely related to a problem with Maven.

I’m pretty sure this actually isn’t a problem anymore. Could you verify by rebuilding Flink and see if the problem remains? Thanks a lot.

Best,

Gordon

On 16 June 2017 at 6:25:10 PM, Tzu-Li (Gordon) Tai ([hidden email]) wrote:

That’s actually what I’m trying to figure out right now, what had changed since then ...

On 16 June 2017 at 12:23:57 PM, Flavio Pompermaier ([hidden email]) wrote:

I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1. Didn't you say that you also had non-shaded guava dependencies in the flink dist jar some days ago?

On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

I was just doing some end-to-end rebuild Flink + cluster execution with ES sink tests, and it seems like the Guava shading problem isn’t there anymore in the flink-dist jar.

On the `release-1.3` branch, built with Maven 3.0.5, the Guava dependencies in flink-dist are all properly shaded.

Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all worked fine without the Guava conflict issue.

So I’m pretty sure that if the problem still exists for you, the conflict would have came from other dependencies in your code.

Cheers,

Gordon

On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi Gordon,

any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio

[Tzu-Li (Gordon) Tai]

Thanks a lot! Please keep me updated with this :)

On 19 June 2017 at 6:33:15 PM, Flavio Pompermaier ([hidden email]) wrote:

Ok, I'll let you know as soon as I recompile Flink 1.3.x.

Thanks,

Flavio

On Mon, Jun 19, 2017 at 7:26 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

It’s most likely related to a problem with Maven.

I’m pretty sure this actually isn’t a problem anymore. Could you verify by rebuilding Flink and see if the problem remains? Thanks a lot.

Best,

Gordon

On 16 June 2017 at 6:25:10 PM, Tzu-Li (Gordon) Tai ([hidden email]) wrote:

That’s actually what I’m trying to figure out right now, what had changed since then ...

On 16 June 2017 at 12:23:57 PM, Flavio Pompermaier ([hidden email]) wrote:

I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1. Didn't you say that you also had non-shaded guava dependencies in the flink dist jar some days ago?

On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

I was just doing some end-to-end rebuild Flink + cluster execution with ES sink tests, and it seems like the Guava shading problem isn’t there anymore in the flink-dist jar.

On the `release-1.3` branch, built with Maven 3.0.5, the Guava dependencies in flink-dist are all properly shaded.

Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all worked fine without the Guava conflict issue.

So I’m pretty sure that if the problem still exists for you, the conflict would have came from other dependencies in your code.

Cheers,

Gordon

On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi Gordon,

any news on this?

On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

This seems like a shading problem then.

I’ve tested this again with Maven 3.0.5, even without building against CDH Hadoop binaries the flink-dist jar contains non-shaded Guava dependencies.

Let me investigate a bit and get back to this!

Cheers,

Gordon

On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier ([hidden email]) wrote:

On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local repo I did:

1. git clone https://github.com/apache/flink.git && cd flink && git checkout tags/release-1.2.1
   
2. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
3. cd flink-dist
4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos
   
5. jar tf target/flink-1.2.1-bin/flink-1.2.1/lib/flink-dist_2.10-1.2.1.jar  | grep MoreExecutors

And I still see guava dependencies:

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/hadoop/shaded/com/google/common/util/concurrent/MoreExecutors.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$2.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$3.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$4.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application$1.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$Application.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutor.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$DirectExecutorService.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

org/apache/flink/shaded/com/google/common/util/concurrent/MoreExecutors.class

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

It seems that it mix togheter guava 11 (probably coming from CDH dependencies) and guava 18 classes.

Also using maven 3.0.5 lead to the same output :(

Best,

Flavio 

On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Yes, those should not be in the flink-dist jar, so the root reason should be that the shading isn’t working properly for your custom build.

If possible, could you try building Flink again with a lower Maven version as specified in the doc, and see if that works?

If so, it could be that Maven 3.3.x simply isn’t shading properly even with the double compilation trick.

On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier ([hidden email]) wrote:

What I did was to take the sources of the new ES connector and I took them into my code.

Flink was compiled with maven 3.3+ but I did the double compilation as specified in the Flink build section.

In flink dist I see guava classes, e.g.:

com/google/common/util/concurrent/MoreExecutors$1.class

com/google/common/util/concurrent/MoreExecutors$SameThreadExecutorService.class

com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors$ScheduledListeningDecorator.class

com/google/common/util/concurrent/MoreExecutors.class

Is it a problem of the shading with Maven 3.3+?

Best,

Flavio

On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES connector).

Another thing to check, if you built Flink yourself, make sure you’re not using Maven 3.3+. There are shading problems when Flink is built with Maven versions higher then that.

The flink-dist jar should not contain any non-shaded Guava dependencies, could you also quickly check that?

On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier ([hidden email]) wrote:

I shaded the Elasticsearch dependency [1] and now the job works.

So I cannot run a job that needs guava 18 on Flink 1.2.1...

[1]  https://www.elastic.co/blog/to-shade-or-not-to-shade

On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Flavio,

Could there be another dependency in your job that requires a conflicting version (w.r.t. ES 2.4.1) of Guava?

I’ve just double checked the flink-dist jar, there doesn’t seem to be any non-shaded Guava dependencies there, so the conflict should not have been caused by Flink.

Cheers,

Gordon

On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ([hidden email]) wrote:

Hi to all,

I'm trying to use the new ES connector to index data from Flink (with ES 2.4.1).

When I try to run it from Eclipse everything is ok, when I run it from the cluster I get the following exception:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
        at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)

In my fat jar there are the classes of guava 18 (ES requires that version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I think that there's guava 11 classes while in flink-hadoop-compatibility there are shade guava 18 dependencies.

How can I make the job successfully run on the cluster?

Best,

Flavio