Advancing watermark with low-traffic Kafka partitions

classic Classic list List threaded Threaded
2 messages Options
Dan
Reply | Threaded
Open this post in threaded view
|

Advancing watermark with low-traffic Kafka partitions

Dan
Hi.

I remember listening to Flink Forward talks about strategies for handling low-traffic Kafka partitions.  One of the talks showed a technique for sending events to advance the watermarks through most levels of the event flow (Kafka source through the sinks).

One of the talks mentioned there were discussions of improving the handling of low-traffic Kafka partitions directly inside of Flink.  What's the status of this?

Thanks!
- Dan
Reply | Threaded
Open this post in threaded view
|

Re: Advancing watermark with low-traffic Kafka partitions

Aeden Jameson
One way to deal with this is by using the source idleness setting. 

On Sat, Jun 19, 2021 at 12:06 PM Dan Hill <[hidden email]> wrote:
Hi.

I remember listening to Flink Forward talks about strategies for handling low-traffic Kafka partitions.  One of the talks showed a technique for sending events to advance the watermarks through most levels of the event flow (Kafka source through the sinks).

One of the talks mentioned there were discussions of improving the handling of low-traffic Kafka partitions directly inside of Flink.  What's the status of this?

Thanks!
- Dan