Re: Kinesis connector classpath issue when running Flink 1.1-SNAPSHOT on YARN

Posted by Tzu-Li Tai on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Kinesis-connector-classpath-issue-when-running-Flink-1-1-SNAPSHOT-on-YARN-tp7611p7617.html

Hi Josh,

I’m looking into the problem. Seems like the connector is somehow using older versions of httpclient.

Can you print the loaded class path at runtime, and check the path & version of the loaded httpclient / httpcore dependency?

i.e. `classOf[HttpConnectionParams].getResource("HttpConnectionParams.class").toString`

Also, on which commit was your kinesis connector built?

Regards,

Gordon

On June 17, 2016 at 1:08:37 AM, Josh ([hidden email]) wrote:

Hey,

I've been running the Kinesis connector successfully now for a couple of weeks, on a Flink cluster running Flink 1.0.3 on EMR 2.7.1/YARN.

Today I've been trying to get it working on a cluster running the current Flink master (1.1-SNAPSHOT) but am running into a classpath issue when starting the job. This only happens when running on EMR/YARN (it's fine when running 1.1-SNAPSHOT locally, and when running 1.0.3 on EMR)

----

 The program finished with the following exception:

java.lang.NoSuchMethodError: org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V

at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96)

at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:187)

at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:136)

at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)

at com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:157)

at com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:137)

at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.<init>(KinesisProxy.java:76)

at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.<init>(FlinkKinesisConsumer.java:166)

at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.<init>(FlinkKinesisConsumer.java:140)

at org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.<init>(FlinkKinesisConsumer.java:123)

---

Any ideas what's going on? 

The job I'm deploying has httpclient 4.3.6 and httpcore 4.3.3 which I believe are the libraries with the HttpConnectionParams class.

Thanks,

Josh