How Flink prioritise read from kafka topics and partitions ?

[sohimankotia]

Hi,

Let's say I have flink Kafka consumer read from 3 topics ,  [ T-1 ,T-2,T-3 ]
.

- T1 and T2 are having partitions equal to 100
- T3 is having partitions equal to 60
- Flink Task (parallelism is 50)

How flink will prioritize Kafka topic ?

If T-3 has more lag than other topics will flink give higher priority to T-3
?


Thanks and Regards
Sohi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Tzu-Li (Gordon) Tai]

Hi Sohi!

On Wed, Jan 23, 2019 at 9:01 PM sohimankotia <[hidden email]> wrote:

Hi,

Let's say I have flink Kafka consumer read from 3 topics ,  [ T-1 ,T-2,T-3 ]
.

- T1 and T2 are having partitions equal to 100
- T3 is having partitions equal to 60
- Flink Task (parallelism is 50)

There isn't any means of prioritizing some partitions / topics to be read first before others in the Flink Kafka Consumer.

 

How flink will prioritize Kafka topic ?

If T-3 has more lag than other topics will flink give higher priority to T-3

No. However, there is some relevant discussion that would be helpful for these situations. Please see [1].
Long story short, if sources are aligned by event-time, the partitions that are lagging behind (in event-time) will slow down other source subtasks until the lagging partition has caught up.
This is a feature that is still in discussion.

Cheers,
Gordon

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952.html

 

?


Thanks and Regards
Sohi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/