Watermarks in Event Time windowing

[Taher Koitawala]

Hi All,

          Can someone show a good example of how watermarks need to be generated when using EventTime windows?  What happens if the watermark is same as the timestamp?  How does the watermark help in the window to be triggered and what if watermarks are kept behind the currentTimestamps in the assignTimestampsAndWatermarks function. 

[vino yang]

Hi Taher,

For some questions, I suggest you read the documentation related to Flink EventTime first, for example [1]

About this question:

What happens if the watermark is same as the timestamp?

Here "timestamp", do you mean the current timestamp of Processing time? If that's the best, it's an ideal state, and it's often difficult to achieve this state, because eventtime always has more or less delay as events are transmitted from the source to the processing system.

Thanks, vino.

[1]:https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/event_timestamps_watermarks.html

Taher Koitawala <[hidden email]> 于2018年9月14日周五 上午2:07写道：

Hi All,

          Can someone show a good example of how watermarks need to be generated when using EventTime windows?  What happens if the watermark is same as the timestamp?  How does the watermark help in the window to be triggered and what if watermarks are kept behind the currentTimestamps in the assignTimestampsAndWatermarks function. 

[Taher Koitawala]

Yes in many cases what we have faced that let's say in a keyed stream an element of a specific key comes in which triggers a new window. If a corresponding elements of the same key does not arrive a new watermark is not generated for the window to purge. Then we faced issues with flink keeping records in the window and not purging them at all. For this reason we explicitly had to write some logic to generate a timestamp this is greater than the currentTimestamp and then forcefully make the window purge its elements. 

On 14-Sep-2018 7:48 AM, "vino yang" <[hidden email]> wrote:

Hi Taher,

For some questions, I suggest you read the documentation related to Flink EventTime first, for example [1]

About this question:

What happens if the watermark is same as the timestamp?

Here "timestamp", do you mean the current timestamp of Processing time? If that's the best, it's an ideal state, and it's often difficult to achieve this state, because eventtime always has more or less delay as events are transmitted from the source to the processing system.

Thanks, vino.

[1]:https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/event_timestamps_watermarks.html

Taher Koitawala <[hidden email]> 于2018年9月14日周五 上午2:07写道：

Hi All,

          Can someone show a good example of how watermarks need to be generated when using EventTime windows?  What happens if the watermark is same as the timestamp?  How does the watermark help in the window to be triggered and what if watermarks are kept behind the currentTimestamps in the assignTimestampsAndWatermarks function. 

[alpinegizmo]

To clarify one thing: keep in mind that Flink does not support per-key watermarks. Watermarks are typically generated per-source, or in the case of kafka, can be per-partition. An idle source (or in the case of kafka, an idle partition) can prevent an event-time window from being triggered, but you don't need to worry about this on a per-key basis.

On Fri, Sep 14, 2018 at 6:08 AM Taher Koitawala <[hidden email]> wrote:

Yes in many cases what we have faced that let's say in a keyed stream an element of a specific key comes in which triggers a new window. If a corresponding elements of the same key does not arrive a new watermark is not generated for the window to purge. Then we faced issues with flink keeping records in the window and not purging them at all. For this reason we explicitly had to write some logic to generate a timestamp this is greater than the currentTimestamp and then forcefully make the window purge its elements. 

On 14-Sep-2018 7:48 AM, "vino yang" <[hidden email]> wrote:

Hi Taher,

For some questions, I suggest you read the documentation related to Flink EventTime first, for example [1]

About this question:

What happens if the watermark is same as the timestamp?

Here "timestamp", do you mean the current timestamp of Processing time? If that's the best, it's an ideal state, and it's often difficult to achieve this state, because eventtime always has more or less delay as events are transmitted from the source to the processing system.

Thanks, vino.

[1]:https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/event_timestamps_watermarks.html

Taher Koitawala <[hidden email]> 于2018年9月14日周五 上午2:07写道：

Hi All,

          Can someone show a good example of how watermarks need to be generated when using EventTime windows?  What happens if the watermark is same as the timestamp?  How does the watermark help in the window to be triggered and what if watermarks are kept behind the currentTimestamps in the assignTimestampsAndWatermarks function. 

--