Re: How to check is Kafka partition "idle" in emitRecordsWithTimestamps

Posted by Till Rohrmann on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/How-to-check-is-Kafka-partition-idle-in-emitRecordsWithTimestamps-tp44048p44099.html

Hi Alexey,

looking at KafkaTopicPartitionStatus, it looks that it does not contain this information. In a nutshell, what you probably have to do is to aggregate the watermarks across all partitions and then pause the consumption of a partition if its watermark advances too much wrt to the minimum watermark. However, this will stall the whole reading process if there is a partition which has no more data. Hence, you will probably also need a mechanism to advance the watermark if the partition becomes idle.

Note that the community is currently working on a new KafkaConnector based on Flink's new source API (FLIP-27). If I am not mistaken, then these new interfaces should eventually also support event time alignment.

Cheers,

Till

On Fri, May 28, 2021 at 7:17 PM Alexey Trenikhun <[hidden email]> wrote:

Hello,

I'm thinking about implementing custom Kafka connector which provides event alignment (similar to FLINK-10921, which seems abandoned). What is the way to determine is partition is idle from override of AbstractFetcher.emitRecordsWithTimestamps()? Does KafkaTopicPartitionState has this information ?

Thanks,

Alexey