IOException when trying flink-twitter example

[ram kumar]

Hi,

Got following exception in taskmanager logs when running TwitterFilterSourceExample,

08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient failed to establish connection properly
08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Done processing, preparing to close connection
08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

Anyone faced this same issue?

Thanks

[Maximilian Michels]

Hi Ram,

This is an Exception thrown in the Twitter client. Unfortunately the
exception is caught and not printed to the screen.

I would suggest to set a breakpoint in the ClientBase to find out
about the reason for the failed connection.

Cheers,
Max

On Thu, Feb 11, 2016 at 9:30 AM, ram kumar <[hidden email]> wrote:

> Hi,
>
> Got following exception in taskmanager logs when running
> TwitterFilterSourceExample,
>
> 08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient failed to establish connection properly
> 08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Done processing, preparing to close connection
> 08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Establishing a connection
> 08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient IOException caught when establishing connection to
> https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
>
>
>
> Anyone faced this same issue?
>
> Thanks

[ram kumar]

org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Initializing Twitter Streaming API connection
12:27:32,134 INFO  com.twitter.hbc.httpclient.BasicClient                        - New connection executed: twitterSourceClient, endpoint: /1.1/statuses/filter.json?delimited=length
12:27:32,135 INFO  org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Twitter Streaming API connection established successfully
12:27:32,233 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
12:27:32,296 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

When i run the job, at first the connection is established,

then this log appears

08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient failed to establish connection properly
08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Done processing, preparing to close connection
08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

On Thu, Feb 11, 2016 at 2:55 PM, Maximilian Michels <[hidden email]> wrote:

Hi Ram,

This is an Exception thrown in the Twitter client. Unfortunately the
exception is caught and not printed to the screen.

I would suggest to set a breakpoint in the ClientBase to find out
about the reason for the failed connection.

Cheers,
Max

On Thu, Feb 11, 2016 at 9:30 AM, ram kumar <[hidden email]> wrote:
> Hi,
>
> Got following exception in taskmanager logs when running
> TwitterFilterSourceExample,
>
> 08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient failed to establish connection properly
> 08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Done processing, preparing to close connection
> 08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Establishing a connection
> 08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient IOException caught when establishing connection to
> https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
>
>
>
> Anyone faced this same issue?
>
> Thanks

[Stephan Ewen]

Looks like an issue with the Twitter Client.

Maybe the log reveals more that can help you figure out what is happening (loss of connection, etc).

On Mon, Feb 15, 2016 at 1:32 PM, ram kumar <[hidden email]> wrote:

org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Initializing Twitter Streaming API connection
12:27:32,134 INFO  com.twitter.hbc.httpclient.BasicClient                        - New connection executed: twitterSourceClient, endpoint: /1.1/statuses/filter.json?delimited=length
12:27:32,135 INFO  org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Twitter Streaming API connection established successfully
12:27:32,233 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
12:27:32,296 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

When i run the job, at first the connection is established,

then this log appears

08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient failed to establish connection properly
08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Done processing, preparing to close connection
08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

On Thu, Feb 11, 2016 at 2:55 PM, Maximilian Michels <[hidden email]> wrote:

Hi Ram,

This is an Exception thrown in the Twitter client. Unfortunately the
exception is caught and not printed to the screen.

I would suggest to set a breakpoint in the ClientBase to find out
about the reason for the failed connection.

Cheers,
Max

On Thu, Feb 11, 2016 at 9:30 AM, ram kumar <[hidden email]> wrote:
> Hi,
>
> Got following exception in taskmanager logs when running
> TwitterFilterSourceExample,
>
> 08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient failed to establish connection properly
> 08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Done processing, preparing to close connection
> 08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Establishing a connection
> 08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient IOException caught when establishing connection to
> https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
>
>
>
> Anyone faced this same issue?
>
> Thanks

[ram kumar]

tried running the TwitterFilterSourceExample as java application, works fine.

facing issue when submitting as flink. looking into flink libraries

On Wed, Feb 17, 2016 at 8:22 PM, Stephan Ewen <[hidden email]> wrote:

Looks like an issue with the Twitter Client.

Maybe the log reveals more that can help you figure out what is happening (loss of connection, etc).

On Mon, Feb 15, 2016 at 1:32 PM, ram kumar <[hidden email]> wrote:

org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Initializing Twitter Streaming API connection
12:27:32,134 INFO  com.twitter.hbc.httpclient.BasicClient                        - New connection executed: twitterSourceClient, endpoint: /1.1/statuses/filter.json?delimited=length
12:27:32,135 INFO  org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Twitter Streaming API connection established successfully
12:27:32,233 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
12:27:32,296 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

When i run the job, at first the connection is established,

then this log appears

08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient failed to establish connection properly
08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Done processing, preparing to close connection
08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

On Thu, Feb 11, 2016 at 2:55 PM, Maximilian Michels <[hidden email]> wrote:

Hi Ram,

This is an Exception thrown in the Twitter client. Unfortunately the
exception is caught and not printed to the screen.

I would suggest to set a breakpoint in the ClientBase to find out
about the reason for the failed connection.

Cheers,
Max

On Thu, Feb 11, 2016 at 9:30 AM, ram kumar <[hidden email]> wrote:
> Hi,
>
> Got following exception in taskmanager logs when running
> TwitterFilterSourceExample,
>
> 08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient failed to establish connection properly
> 08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Done processing, preparing to close connection
> 08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Establishing a connection
> 08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient IOException caught when establishing connection to
> https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
>
>
>
> Anyone faced this same issue?
>
> Thanks

[igor.berman]

http://stackoverflow.com/questions/32379590/ioexcpetion-while-connecting-to-twitter-streaming-api-with-apache-flink

there is a problem with http client, 

I've solved it by downloading httpclient-4.2.6.jar jar into flink/lib and renamed it to have "a" (ahttpclient-4.2.6.jar) so it will be added to classpath of flink runtime first(by bin/config.sh)

On 21 February 2016 at 22:56, ram kumar <[hidden email]> wrote:

tried running the TwitterFilterSourceExample as java application, works fine.

facing issue when submitting as flink. looking into flink libraries

On Wed, Feb 17, 2016 at 8:22 PM, Stephan Ewen <[hidden email]> wrote:

Looks like an issue with the Twitter Client.

Maybe the log reveals more that can help you figure out what is happening (loss of connection, etc).

On Mon, Feb 15, 2016 at 1:32 PM, ram kumar <[hidden email]> wrote:

org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Initializing Twitter Streaming API connection
12:27:32,134 INFO  com.twitter.hbc.httpclient.BasicClient                        - New connection executed: twitterSourceClient, endpoint: /1.1/statuses/filter.json?delimited=length
12:27:32,135 INFO  org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Twitter Streaming API connection established successfully
12:27:32,233 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
12:27:32,296 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

When i run the job, at first the connection is established,

then this log appears

08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient failed to establish connection properly
08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Done processing, preparing to close connection
08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

On Thu, Feb 11, 2016 at 2:55 PM, Maximilian Michels <[hidden email]> wrote:

Hi Ram,

This is an Exception thrown in the Twitter client. Unfortunately the
exception is caught and not printed to the screen.

I would suggest to set a breakpoint in the ClientBase to find out
about the reason for the failed connection.

Cheers,
Max

On Thu, Feb 11, 2016 at 9:30 AM, ram kumar <[hidden email]> wrote:
> Hi,
>
> Got following exception in taskmanager logs when running
> TwitterFilterSourceExample,
>
> 08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient failed to establish connection properly
> 08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Done processing, preparing to close connection
> 08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Establishing a connection
> 08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient IOException caught when establishing connection to
> https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
>
>
>
> Anyone faced this same issue?
>
> Thanks

[Stephan Ewen]

Hi!

Thanks for figuring this out.

Turns out that this issue was caused by a conflict between versions of the Apache HTTP Components library used by the Twitter Client, and used by the Hadoop/HDFS libraries.

From version 1.0 on, we shade that library in Hadoop, so that issue should be solved and the example work without this  workaround.

Stephan

On Sun, Feb 28, 2016 at 11:46 AM, Igor Berman <[hidden email]> wrote:

http://stackoverflow.com/questions/32379590/ioexcpetion-while-connecting-to-twitter-streaming-api-with-apache-flink

there is a problem with http client, 

I've solved it by downloading httpclient-4.2.6.jar jar into flink/lib and renamed it to have "a" (ahttpclient-4.2.6.jar) so it will be added to classpath of flink runtime first(by bin/config.sh)

On 21 February 2016 at 22:56, ram kumar <[hidden email]> wrote:

tried running the TwitterFilterSourceExample as java application, works fine.

facing issue when submitting as flink. looking into flink libraries

On Wed, Feb 17, 2016 at 8:22 PM, Stephan Ewen <[hidden email]> wrote:

Looks like an issue with the Twitter Client.

Maybe the log reveals more that can help you figure out what is happening (loss of connection, etc).

On Mon, Feb 15, 2016 at 1:32 PM, ram kumar <[hidden email]> wrote:

org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Initializing Twitter Streaming API connection
12:27:32,134 INFO  com.twitter.hbc.httpclient.BasicClient                        - New connection executed: twitterSourceClient, endpoint: /1.1/statuses/filter.json?delimited=length
12:27:32,135 INFO  org.apache.flink.streaming.connectors.twitter.TwitterFilterSource  - Twitter Streaming API connection established successfully
12:27:32,233 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
12:27:32,296 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

When i run the job, at first the connection is established,

then this log appears

08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient failed to establish connection properly
08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Done processing, preparing to close connection
08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient Establishing a connection
08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase                         - twitterSourceClient IOException caught when establishing connection to https://stream.twitter.com/1.1/statuses/filter.json?delimited=length

On Thu, Feb 11, 2016 at 2:55 PM, Maximilian Michels <[hidden email]> wrote:

Hi Ram,

This is an Exception thrown in the Twitter client. Unfortunately the
exception is caught and not printed to the screen.

I would suggest to set a breakpoint in the ClientBase to find out
about the reason for the failed connection.

Cheers,
Max

On Thu, Feb 11, 2016 at 9:30 AM, ram kumar <[hidden email]> wrote:
> Hi,
>
> Got following exception in taskmanager logs when running
> TwitterFilterSourceExample,
>
> 08:25:19,207 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient failed to establish connection properly
> 08:25:19,207 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Done processing, preparing to close connection
> 08:25:19,208 INFO  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient Establishing a connection
> 08:25:19,225 WARN  com.twitter.hbc.httpclient.ClientBase
> - twitterSourceClient IOException caught when establishing connection to
> https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
>
>
>
> Anyone faced this same issue?
>
> Thanks