kafka consumers partition count and parallelism

[Prashant Deva]

Can a Kafka Consumer Source have more tasks run in parallel than the number of partitions for the topic it is the source of? Or is the max parallelism of the source constrained by max partitions of the topic?

[rmetzger0]

Hey Prashant,

the Kafka Consumer parallelism is constrained by the number of partitions the topic(s) have. If you have configured the Kafka Consumer in Flink with a parallelism of 100, but your topic has only 20 partitions, 80 consumer instances in Flink will be idle.

On Mon, Apr 26, 2021 at 2:54 AM Prashant Deva <[hidden email]> wrote:

Can a Kafka Consumer Source have more tasks run in parallel than the number of partitions for the topic it is the source of? Or is the max parallelism of the source constrained by max partitions of the topic?