Question about the need of consumer groups from kafka

[Ricardo Dinis]

Hi,

If flink can manage kafka without using its own's mechanism (kafka consumers) it still need the consumer group defined from kafka?

And why the examples is needed to use the plugin for shadding creating a fat-jar?

Thanks

[Till Rohrmann]

Hi Ricardo,

could you please give a bit more details what you mean with "not using its own mechanism"? Flink's Kafka connector uses the Kafka consumer and producer (to some extent) API to talk to Kafka. The consumer groups are a central concept of Kafka and as such, the Flink Kafka connector has to know it in order to consume from the right group.

Creating a fat-jar will make sure that the jar includes additional dependencies such as the flink-kafka-connector which is not present on the Flink cluster.

Cheers,

Till

On Fri, Feb 16, 2018 at 2:05 PM, Ricardo Dinis <[hidden email]> wrote:

Hi,

If flink can manage kafka without using its own's mechanism (kafka consumers) it still need the consumer group defined from kafka?

And why the examples is needed to use the plugin for shadding creating a fat-jar?

Thanks