NoClassDefFoundError on cluster with httpclient 4.5.2

[Yassine MARZOUGUI]

Hi all,

I'm using httpclient with the following dependency:

<dependency>

<groupId>org.apache.httpcomponents</groupId>

<artifactId>httpclient</artifactId>

<version>4.5.2</version>

</dependency>

On local mode, the program works correctly, but when executed on the cluster, I get the following exception:

java.lang.Exception: The user defined 'open(Configuration)' method in class org.myorg.quickstart.Frequencies$2 caused an exception: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1337)

        at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.openTask(ChainedFlatMapDriver.java:47)

        at org.apache.flink.runtime.operators.BatchTask.openChainedTasks(BatchTask.java:1377)

        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:124)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:966)

        at org.myorg.quickstart.Frequencies$2.open(Frequencies.java:82)

        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1335)

        ... 5 more

I'm using Flink 1.1.3. Any idea how to solve the problem? Thank you.

Best,

Yassine

[Till Rohrmann]

Hi Yassine,

can you check whether the httpclient jar is contained in your job jar which you submit to the cluster?

Cheers,

Till

On Wed, Oct 19, 2016 at 6:41 PM, Yassine MARZOUGUI <[hidden email]> wrote:

Hi all,

I'm using httpclient with the following dependency:

<dependency>

<groupId>org.apache.httpcomponents</groupId>

<artifactId>httpclient</artifactId>

<version>4.5.2</version>

</dependency>

On local mode, the program works correctly, but when executed on the cluster, I get the following exception:

java.lang.Exception: The user defined 'open(Configuration)' method in class org.myorg.quickstart.Frequencies$2 caused an exception: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1337)

        at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.openTask(ChainedFlatMapDriver.java:47)

        at org.apache.flink.runtime.operators.BatchTask.openChainedTasks(BatchTask.java:1377)

        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:124)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:966)

        at org.myorg.quickstart.Frequencies$2.open(Frequencies.java:82)

        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1335)

        ... 5 more

I'm using Flink 1.1.3. Any idea how to solve the problem? Thank you.

Best,

Yassine

[Yassine MARZOUGUI]

Hi Till,

The httpclient jar is included in the job jar. Looking at a similar issue FLINK-4587, It turns out the problem is with maven shade plugin, since I'm building Flink from sources with maven 3.3.x.

I was able to solve the problem by rebuilding "flink-dist" as suggested by Stephan in the comments.

Best,

Yassine

2016-10-20 11:05 GMT+02:00 Till Rohrmann <[hidden email]>:

Hi Yassine,

can you check whether the httpclient jar is contained in your job jar which you submit to the cluster?

Cheers,

Till

On Wed, Oct 19, 2016 at 6:41 PM, Yassine MARZOUGUI <[hidden email]> wrote:

Hi all,

I'm using httpclient with the following dependency:

<dependency>

<groupId>org.apache.httpcomponents</groupId>

<artifactId>httpclient</artifactId>

<version>4.5.2</version>

</dependency>

On local mode, the program works correctly, but when executed on the cluster, I get the following exception:

java.lang.Exception: The user defined 'open(Configuration)' method in class org.myorg.quickstart.Frequencies$2 caused an exception: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1337)

        at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.openTask(ChainedFlatMapDriver.java:47)

        at org.apache.flink.runtime.operators.BatchTask.openChainedTasks(BatchTask.java:1377)

        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:124)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:966)

        at org.myorg.quickstart.Frequencies$2.open(Frequencies.java:82)

        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1335)

        ... 5 more

I'm using Flink 1.1.3. Any idea how to solve the problem? Thank you.

Best,

Yassine

[Till Rohrmann]

Great to hear that you solved your problem :-)

Cheers,

Till

On Fri, Oct 21, 2016 at 12:34 PM, Yassine MARZOUGUI <[hidden email]> wrote:

Hi Till,

The httpclient jar is included in the job jar. Looking at a similar issue FLINK-4587, It turns out the problem is with maven shade plugin, since I'm building Flink from sources with maven 3.3.x.

I was able to solve the problem by rebuilding "flink-dist" as suggested by Stephan in the comments.

Best,

Yassine

2016-10-20 11:05 GMT+02:00 Till Rohrmann <[hidden email]>:

Hi Yassine,

can you check whether the httpclient jar is contained in your job jar which you submit to the cluster?

Cheers,

Till

On Wed, Oct 19, 2016 at 6:41 PM, Yassine MARZOUGUI <[hidden email]> wrote:

Hi all,

I'm using httpclient with the following dependency:

<dependency>

<groupId>org.apache.httpcomponents</groupId>

<artifactId>httpclient</artifactId>

<version>4.5.2</version>

</dependency>

On local mode, the program works correctly, but when executed on the cluster, I get the following exception:

java.lang.Exception: The user defined 'open(Configuration)' method in class org.myorg.quickstart.Frequencies$2 caused an exception: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1337)

        at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.openTask(ChainedFlatMapDriver.java:47)

        at org.apache.flink.runtime.operators.BatchTask.openChainedTasks(BatchTask.java:1377)

        at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:124)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory

        at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:966)

        at org.myorg.quickstart.Frequencies$2.open(Frequencies.java:82)

        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)

        at org.apache.flink.runtime.operators.BatchTask.openUserCode(BatchTask.java:1335)

        ... 5 more

I'm using Flink 1.1.3. Any idea how to solve the problem? Thank you.

Best,

Yassine