Advancing watermark with low-traffic Kafka partitions

[Dan]

Hi.

I remember listening to Flink Forward talks about strategies for handling low-traffic Kafka partitions.  One of the talks showed a technique for sending events to advance the watermarks through most levels of the event flow (Kafka source through the sinks).

One of the talks mentioned there were discussions of improving the handling of low-traffic Kafka partitions directly inside of Flink.  What's the status of this?

Thanks!

- Dan

[Aeden Jameson]

One way to deal with this is by using the source idleness setting. 

https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources

On Sat, Jun 19, 2021 at 12:06 PM Dan Hill <[hidden email]> wrote:

Hi.

I remember listening to Flink Forward talks about strategies for handling low-traffic Kafka partitions.  One of the talks showed a technique for sending events to advance the watermarks through most levels of the event flow (Kafka source through the sinks).

One of the talks mentioned there were discussions of improving the handling of low-traffic Kafka partitions directly inside of Flink.  What's the status of this?

Thanks!

- Dan