How Flink distinguishes between late and in-time events?

[orips]

In the documentation it states that:

[…], Flink keeps the state of windows until their allowed lateness expires. Once this happens, Flink removes the window and deletes its state, as also described in the Window Lifecycle section.

However, something doesn't make sense to me.

If Flink deletes the window state, then how can it know that subsequent events are late? i.e. if the state is deleted, then Flink has no way of knowing than an event is late, because it can think it's just a new event, unless it keeps track of which keyed windows are closed forever.

Does Flink remember which keyed windows are closed forever?

Thanks.

[Piotr Nowojski-4]

Hi Ori,

No. Flink does it differently. Operators that are keeping track of late events, are remembering the latest watermark. If a new element arrives with even time lower compared to the latest watermark, it is marked as a late event [1]

Piotrek

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/timely-stream-processing.html#lateness

czw., 20 sie 2020 o 17:13 Ori Popowski <[hidden email]> napisał(a):

In the documentation it states that:

[…], Flink keeps the state of windows until their allowed lateness expires. Once this happens, Flink removes the window and deletes its state, as also described in the Window Lifecycle section.

However, something doesn't make sense to me.

If Flink deletes the window state, then how can it know that subsequent events are late? i.e. if the state is deleted, then Flink has no way of knowing than an event is late, because it can think it's just a new event, unless it keeps track of which keyed windows are closed forever.

Does Flink remember which keyed windows are closed forever?

Thanks.

[orips]

That makes sense. Thanks

On Thu, Aug 20, 2020 at 7:45 PM Piotr Nowojski <[hidden email]> wrote:

Hi Ori,

No. Flink does it differently. Operators that are keeping track of late events, are remembering the latest watermark. If a new element arrives with even time lower compared to the latest watermark, it is marked as a late event [1]

Piotrek

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/timely-stream-processing.html#lateness

czw., 20 sie 2020 o 17:13 Ori Popowski <[hidden email]> napisał(a):

In the documentation it states that:

[…], Flink keeps the state of windows until their allowed lateness expires. Once this happens, Flink removes the window and deletes its state, as also described in the Window Lifecycle section.

However, something doesn't make sense to me.

If Flink deletes the window state, then how can it know that subsequent events are late? i.e. if the state is deleted, then Flink has no way of knowing than an event is late, because it can think it's just a new event, unless it keeps track of which keyed windows are closed forever.

Does Flink remember which keyed windows are closed forever?

Thanks.

... [show rest of quote]