consume kafka stream with flink

[tanguy racinet]

Hello,

I am currently trying to develop am algorithm mining frequent item sets over a data stream.

I am using kafka to generate the stream, however I cannot manage to link Flink to Kafka.

The code presented here is working but only using Flink version 0.9.1

https://github.com/dataArtisans/kafka-example/blob/master/src/main/java/com/dataartisans/ReadFromKafka.java

It doesn’t work for me when using FLink 10-SNAPSHOT or the latest release.

I tried several other things and was hoping someone could help me understand the differences between kafkaSource, FlinkKafkaConsummer, FlinkKafkaConsummer081 and FlinkKafkaConsummer082.

Which one is the latest API ? Is there another way to consume Kafka steam with Flink that is even more up to date ? Or should I use FlinkKafkaConsummer082 and stay with Flink version 9 ?

Thank you for your help.

Regards,

Tanguy R.

[rmetzger0]

Hi Tanguy,

I would recommend to refer to the documentation of the specific Flink version you are using.

This is the documentation for 1.0-SNAPSHOT: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/connectors/kafka.html

and this is the doc for 0.10.x: https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#apache-kafka

I would recommend to use Flink 0.10.x and the FlinkKafkaConsumer082. With Flink 1.0 the consumer will be renamed to FlinkKafkaConsumer08.

Let me know if you have further questions.

Regards,

Robert

On Fri, Feb 12, 2016 at 12:07 PM, Tanguy Racinet <[hidden email]> wrote:

Hello,

I am currently trying to develop am algorithm mining frequent item sets over a data stream.

I am using kafka to generate the stream, however I cannot manage to link Flink to Kafka.

The code presented here is working but only using Flink version 0.9.1

https://github.com/dataArtisans/kafka-example/blob/master/src/main/java/com/dataartisans/ReadFromKafka.java

It doesn’t work for me when using FLink 10-SNAPSHOT or the latest release.

I tried several other things and was hoping someone could help me understand the differences between kafkaSource, FlinkKafkaConsummer, FlinkKafkaConsummer081 and FlinkKafkaConsummer082.

Which one is the latest API ? Is there another way to consume Kafka steam with Flink that is even more up to date ? Or should I use FlinkKafkaConsummer082 and stay with Flink version 9 ?

Thank you for your help.

Regards,

Tanguy R.