NoClassDefFoundError with ElasticsearchSink on Yarn

[Hausmann, Steffen]

Hi there,

I’m running a flink program that reads from a Kinesis stream and
eventually writes to an Elasticsearch2 sink. When I’m running the
program locally from the IDE, everything seems to work fine, but when
I’m executing the same program on an EMR cluster with Yarn, a
NoClassDefFoundError occurs:

java.lang.NoSuchMethodError:
com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;

        at
org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)

        at
org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:133)

        at
org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.open(ElasticsearchSink.java:164)

        at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

        at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)

        at
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376)

        at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:256)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)

        at java.lang.Thread.run(Thread.java:745)

I’ve installed flink on an EMR cluster from the binary distribution
flink-1.1.1-bin-hadoop27-scala_2.10.tgz. I build the jar file locally
with mvn clean package(I’ve attached the pom.xml for reference) and copy
it to the cluster for execution. There is a thread on this list that
seems to be related, but I’m afraid I couldn’t draw any conclusions from it:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/classpath-issue-on-yarn-tt6442.html#none


Any idea, what’s wrong?

Thanks,

Steffen

[Fabian Hueske-2]

Hi Steffen,

this looks like a Guava version mismatch to me.
Are you running exactly the same program on your local machine or did you add dependencies to run it on the cluster (e.g. Kinesis).

Maybe Kinesis and Elasticsearch are using different Guava versions?

Best, Fabian

2016-09-01 10:45 GMT+02:00 Hausmann, Steffen <[hidden email]>:

Hi there,

I’m running a flink program that reads from a Kinesis stream and eventually writes to an Elasticsearch2 sink. When I’m running the program locally from the IDE, everything seems to work fine, but when I’m executing the same program on an EMR cluster with Yarn, a NoClassDefFoundError occurs:

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;

       at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)

       at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:133)

       at org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.open(ElasticsearchSink.java:164)

       at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

       at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)

       at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376)

       at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:256)

       at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)

       at java.lang.Thread.run(Thread.java:745)

I’ve installed flink on an EMR cluster from the binary distribution flink-1.1.1-bin-hadoop27-scala_2.10.tgz. I build the jar file locally with mvn clean package(I’ve attached the pom.xml for reference) and copy it to the cluster for execution. There is a thread on this list that seems to be related, but I’m afraid I couldn’t draw any conclusions from it:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/classpath-issue-on-yarn-tt6442.html#none


Any idea, what’s wrong?

Thanks,

Steffen