Re: Per Partition Watermarking source idleness

Posted by prakhar_mathur on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Per-Partition-Watermarking-source-idleness-tp29596p29638.html

Hi,

Thanks for the response. Can you point me to some examples of such strategy?

On Sat, Aug 24, 2019, 06:01 Eduardo Winpenny Tejedor <[hidden email]> wrote:

Hi Prakhar,

Everything is probably working as expected, if a partition does not receive any messages then the watermark of the operator does not advance (as it is the minimum across all partitions).

You'll need to define a strategy for the watermark to advance even when no messages are received for a particular partition.

Regards,

Eduardo

On Fri, 23 Aug 2019, 10:35 Prakhar Mathur, <[hidden email]> wrote:

Hi,

We are using flink v1.6. We are facing data loss issues while consuming data from older offsets in Kafka with windowing. We are exploring per partition watermarking strategy. But we noticed that when we are trying to consume from multiple topics and if any of the partition is not receiving data it just blocks everything. Do we have a known solution for this?