Tweets Stream input for Flink

[Mustafa Elbehery]

Hi All, 

We have created a Java model for the Twitter API, and are looking for a way to integrate it with Flink.

aalexandrov[8:44 PM]We have a sequence of tweets serialized as JSON objects and need to parse them accordingly in order to convert them to our Java model.

aalexandrov[8:45 PM]Is there an InputReader that can parse a JSON array into a sequence of strings (each string representing a single array element)?
Regards.

--

Mustafa Elbehery

EIT ICT Labs Master School

[phone]
skype: mustafaelbehery87

[Márton Balassi]

Dear Mustafa,

Flink has built-in support for stream processing providing connectors for a number of input sources, the Twitter Streaming API amongst others. We provide an example of reading from the publicly available tweets and parsing the flat JSON objects to build our internal DataStream representation containing the relevant fields of the JSON.

Documentation:

http://flink.incubator.apache.org/docs/0.7-incubating/streaming_guide.html#twitter-streaming-api

Relevant example code:

https://github.com/apache/incubator-flink/blob/master/flink-addons/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/twitter/TwitterLocal.java

Operator parsing the JSON into a DataStream:

https://github.com/apache/incubator-flink/blob/master/flink-addons/flink-streaming/flink-streaming-connectors/src/main/java/org/apache/flink/streaming/connectors/json/JSONParseFlatMap.java

Cheers,

Marton

On Tue, Oct 28, 2014 at 8:47 PM, Mustafa Elbehery <[hidden email]> wrote:

Hi All, 

We have created a Java model for the Twitter API, and are looking for a way to integrate it with Flink.

aalexandrov[8:44 PM]We have a sequence of tweets serialized as JSON objects and need to parse them accordingly in order to convert them to our Java model.

aalexandrov[8:45 PM]Is there an InputReader that can parse a JSON array into a sequence of strings (each string representing a single array element)?
Regards.

--

Mustafa Elbehery

EIT ICT Labs Master School

<a href="tel:%2B49%280%2916%202819%202148" value="[phone]" target="_blank">[phone]
skype: mustafaelbehery87