Late Events with BoundedOutOfOrdernessTimestampExtractor and allowed lateness

[nico-2]

Hi,

I struggle a bit to understand the difference between BoundedOutOfOrdernessTimestampExtractor and the allowed lateness function of a window...

As I understand it, when I use BoundedOutOfOrdernessTimestampExtractor the watermark is lagging behind the real event time of the stream with maxOutOfOrderness. So for the window trigger late events won't be late as the watermark is lagging.



In Contrast, allowed lateness will wait after the watermark passed. So using allowed lateness a window could fire twice - after the watermark and the arrival of late events.



Is that the only difference? What would happen if I use both? Do I have a double delay?



Does a best practice exist when to BoundedOutOfOrdernessTimestampExtractor or allowed latesness?



Best regards,


Nico  

[Yassine MARZOUGUI]

Hi Nico,

You might check Fabian's answer on a similar question I posted previousely on the mailing list, it can be helpful : 

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/BoundedOutOfOrdernessTimestampExtractor-and-allowedlateness-td9583.html

Best,

Yassine

On Mar 15, 2017 18:58, "Nico" <[hidden email]> wrote:

Hi,

I struggle a bit to understand the difference between BoundedOutOfOrdernessTimestampExtractor and the allowed lateness function of a window...

As I understand it, when I use BoundedOutOfOrdernessTimestampExtractor the watermark is lagging behind the real event time of the stream with maxOutOfOrderness. So for the window trigger late events won't be late as the watermark is lagging.



In Contrast, allowed lateness will wait after the watermark passed. So using allowed lateness a window could fire twice - after the watermark and the arrival of late events.



Is that the only difference? What would happen if I use both? Do I have a double delay?



Does a best practice exist when to BoundedOutOfOrdernessTimestampExtractor or allowed latesness?



Best regards,


Nico  

[nico-2]

Hi Yassine,

I forgot to say thank you for poiting to the post. It was really useful.

Best,

Nico :)

2017-03-15 20:19 GMT+01:00 Yassine MARZOUGUI <[hidden email]>:

Hi Nico,

You might check Fabian's answer on a similar question I posted previousely on the mailing list, it can be helpful : 

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/BoundedOutOfOrdernessTimestampExtractor-and-allowedlateness-td9583.html

Best,

Yassine

On Mar 15, 2017 18:58, "Nico" <[hidden email]> wrote:

Hi,

I struggle a bit to understand the difference between BoundedOutOfOrdernessTimestampExtractor and the allowed lateness function of a window...

As I understand it, when I use BoundedOutOfOrdernessTimestampExtractor the watermark is lagging behind the real event time of the stream with maxOutOfOrderness. So for the window trigger late events won't be late as the watermark is lagging.



In Contrast, allowed lateness will wait after the watermark passed. So using allowed lateness a window could fire twice - after the watermark and the arrival of late events.



Is that the only difference? What would happen if I use both? Do I have a double delay?



Does a best practice exist when to BoundedOutOfOrdernessTimestampExtractor or allowed latesness?



Best regards,


Nico