Flink Watermark and timing

[Björn Zachrisson]

Hi,

I have a question regarding timing of events.

According to;

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/event_time.html#event-time-and-watermarks

All events up to and equal to watermark should be handled in "the prevoius window".

In my case I use event-timestamp.

I'm testing the timing out.

The case is events from 2000-01-01 02:00:00 and up to 2000-01-01 02:20:00 where eavh event is 2 minutes apart. I try to group the events in 5 minute windows

2000-01-01 02:00:00 => 2000-01-01 02:05:00

2000-01-01 02:05:00 => 2000-01-01 02:10:00

2000-01-01 02:10:00 => 2000-01-01 02:15:00

2000-01-01 02:15:00 => 2000-01-01 02:20:00

How ever, events at the exakt time 02:10:00 (946692600000) is put in the Window "2000-01-01 02:10:00 => 2000-01-01 02:15:00" which is not according to what i can read on the wiki. 

This is the exakt result;

2000-01-01 02:00:00, 946692000000

2000-01-01 02:02:00, 946692120000

2000-01-01 02:04:00, 946692240000

2000-01-01 02:06:00, 946692360000

2000-01-01 02:08:00, 946692480000

2000-01-01 02:10:00, 946692600000

2000-01-01 02:12:00, 946692720000

2000-01-01 02:14:00, 946692840000

2000-01-01 02:16:00, 946692960000

2000-01-01 02:18:00, 946693080000

2000-01-01 02:20:00, 946693200000

Is this due to that I'm using event time extractor or what might be the case?

Regards

Björn 

[Timo Walther]

Hi Björn,


I don't know if I get your example correctly, but I think your explanation "All events up to and equal to watermark should be handled in the prevoius window" is not 100% correct. Watermarks just indicate the progress ("until here we have seen all events with lower timestamp than X") and trigger the evaluation of a window. The assignment of events to windows is based on the timestamp not the watermark. The documentation will be improved for the upcoming release:

https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/windows.html#window-assigners

"Time-based windows have a start timestamp (inclusive) and an end timestamp (exclusive) that together describe the size of the window. "

I hope this helps.

Regards,
Timo


Am 10/2/17 um 1:06 PM schrieb Björn Zachrisson:

Hi,

I have a question regarding timing of events.

According to;

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/event_time.html#event-time-and-watermarks

All events up to and equal to watermark should be handled in "the prevoius window".

In my case I use event-timestamp.

I'm testing the timing out.

The case is events from 2000-01-01 02:00:00 and up to 2000-01-01 02:20:00 where eavh event is 2 minutes apart. I try to group the events in 5 minute windows

2000-01-01 02:00:00 => 2000-01-01 02:05:00

2000-01-01 02:05:00 => 2000-01-01 02:10:00

2000-01-01 02:10:00 => 2000-01-01 02:15:00

2000-01-01 02:15:00 => 2000-01-01 02:20:00

How ever, events at the exakt time 02:10:00 (946692600000) is put in the Window "2000-01-01 02:10:00 => 2000-01-01 02:15:00" which is not according to what i can read on the wiki. 

This is the exakt result;

2000-01-01 02:00:00, 946692000000

2000-01-01 02:02:00, 946692120000

2000-01-01 02:04:00, 946692240000

2000-01-01 02:06:00, 946692360000

2000-01-01 02:08:00, 946692480000

2000-01-01 02:10:00, 946692600000

2000-01-01 02:12:00, 946692720000

2000-01-01 02:14:00, 946692840000

2000-01-01 02:16:00, 946692960000

2000-01-01 02:18:00, 946693080000

2000-01-01 02:20:00, 946693200000

Is this due to that I'm using event time extractor or what might be the case?

Regards

Björn