Apache Flink - Question about TriggerResult.FIRE

[M Singh]

Hi Flink Users

I have a few questions about triggers:

If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window.  If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ? 

Secondly, how does the FIRE option affect the sliding window computation ?

If there are any other insights/pitfalls while dealing with this, please let me know.

Thanks

Mans

[M Singh]

Hi Guys

Is there any insight into this ?

Thanks

Mans

On Monday, November 13, 2017 11:19 AM, M Singh <[hidden email]> wrote:

Hi Flink Users

I have a few questions about triggers:

If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window.  If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ? 

Secondly, how does the FIRE option affect the sliding window computation ?

If there are any other insights/pitfalls while dealing with this, please let me know.

Thanks

Mans

[Stefan Richter]

Hi,

I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.

For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.

Best,

Stefan

Am 13.11.2017 um 20:19 schrieb M Singh <[hidden email]>:

Hi Flink Users

I have a few questions about triggers:

If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window.  If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ? 

Secondly, how does the FIRE option affect the sliding window computation ?

If there are any other insights/pitfalls while dealing with this, please let me know.

Thanks

Mans

[Aljoscha Krettek]

Yes, all of this is correct. Sliding windows in fact look like completely separate windows to the windowing system.

Best,

Aljoscha

On 16. Nov 2017, at 10:15, Stefan Richter <[hidden email]> wrote:

Hi,

I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.

For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.

Best,

Stefan

Am 13.11.2017 um 20:19 schrieb M Singh <[hidden email]>:

Hi Flink Users

I have a few questions about triggers:

If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window.  If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ? 

Secondly, how does the FIRE option affect the sliding window computation ?

If there are any other insights/pitfalls while dealing with this, please let me know.

Thanks

Mans

[M Singh]

Thanks Stefan and Aljoscha for your responses.

Stefan - When I mentioned "new window" - I meant the next window being created. 

Eg:  if the event was in w1 based processing time and the trigger returned FIRE - then after the window function is computed, what happens to the events in that window (w1).  Are they (elements in w1) propagated to the next processing time window (w2) ?  If not, then what is the difference between FIRE and FIRE_AND_PURGE and when do we use FIRE vs FIRE_AND PURGE ?  Are there any examples to demonstrated these differences ?

Thanks again for your help. 

Mans

On Thursday, November 16, 2017 5:16 AM, Aljoscha Krettek <[hidden email]> wrote:

Yes, all of this is correct. Sliding windows in fact look like completely separate windows to the windowing system.

Best,

Aljoscha

On 16. Nov 2017, at 10:15, Stefan Richter <[hidden email]> wrote:

Hi,

I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.

For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.

Best,

Stefan

Am 13.11.2017 um 20:19 schrieb M Singh <[hidden email]>:

Hi Flink Users

I have a few questions about triggers:

If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window.  If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ? 

Secondly, how does the FIRE option affect the sliding window computation ?

If there are any other insights/pitfalls while dealing with this, please let me know.

Thanks

Mans

[M Singh]

Also, Stefan - You mentioned 

"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."

I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged.  Am I missing something ?  

Once again, if there are any examples it would greatly help me in understanding the semantics/usage scenarios.

Mans

On Friday, November 17, 2017 10:33 AM, M Singh <[hidden email]> wrote:

Thanks Stefan and Aljoscha for your responses.

Stefan - When I mentioned "new window" - I meant the next window being created. 

Eg:  if the event was in w1 based processing time and the trigger returned FIRE - then after the window function is computed, what happens to the events in that window (w1).  Are they (elements in w1) propagated to the next processing time window (w2) ?  If not, then what is the difference between FIRE and FIRE_AND_PURGE and when do we use FIRE vs FIRE_AND PURGE ?  Are there any examples to demonstrated these differences ?

Thanks again for your help. 

Mans

On Thursday, November 16, 2017 5:16 AM, Aljoscha Krettek <[hidden email]> wrote:

Yes, all of this is correct. Sliding windows in fact look like completely separate windows to the windowing system.

Best,

Aljoscha

On 16. Nov 2017, at 10:15, Stefan Richter <[hidden email]> wrote:

Hi,

I think the effect is pretty straight forward, the elements in a window are not purged if the trigger is only FIRE and not FIRE_AND_PURGE. Unfortunately, your question is a bit unclear about what exactly you mean by „new window“: a truly „new“ window or another triggering of the previous (non-purged) window? In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger.

For sliding windows, if I remember correctly, every slide is actually a different window and elements are just added repeatedly to all windows in which they belong. So window n+1 should not be affected by whether or not window n purges or not. Maybe Aljoscha (in CC) can confirm this for us.

Best,

Stefan

Am 13.11.2017 um 20:19 schrieb M Singh <[hidden email]>:

Hi Flink Users

I have a few questions about triggers:

If a trigger returns TriggerResult.FIRE from say the onProcessingTime method - the window computation is triggered but elements are kept in the window.  If there a second invocation of the onProcessingTime method will the elements from the previous window (which were not purged) a part of the new window computation along with new events added since the last FIRE event ? 

Secondly, how does the FIRE option affect the sliding window computation ?

If there are any other insights/pitfalls while dealing with this, please let me know.

Thanks

Mans

[Stefan Richter]

Hi,

"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."

I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged.  Am I missing something ?  

No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.

Best,

Stefan

[Aljoscha Krettek]

Hi Mans,

For understanding the difference between FIRE and FIRE_AND_PURGE it's helpful to look at the cases where it really makes a difference. In my opinion this only makes a difference when you have event-time windowing and when you have multiple firing for the same window (i.e. multiple firings for the window [12:00,13:00) for a key x). You have this either when you have late data or when you have a custom trigger that can fire speculatively. Let's look at the first case and see where FIRE/FIRE_AND_PURGE act differently.

Assume we have window assigner = "tumbling windows of 10 ms" and allowed lateness 5 ms. We get this data:

event1, ts = 1

watermark, ts = 11

event2, ts = 2

What we get from this is a firing when the watermark arrives with the window contents (event1) for window [0, 9). Then, when event2 arrives we get a second firing (because the default event-time trigger will FIRE when late data arrives for a window) for window [0, 9) with contents (event1, event2) if our trigger returns FIRE or with contents (event2) if our trigger returns FIRE_AND_PURGE.

If we had allowed lateness 0 ms we wouldn't have the second firing for the late event and we wouldn't see a difference between FIRE and FIRE_AND_PURGE.

This can be extended to the second case of a custom trigger that might fire early (based on the count of events in the window or processing time).

I hope this helps.

Aljoscha

On 20. Nov 2017, at 11:18, Stefan Richter <[hidden email]> wrote:

Hi,

"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."

I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged.  Am I missing something ?  

No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.

Best,

Stefan

[M Singh]

Hi:

Another question - what happens if the trigger never calls PURGE or FIRE_AND_PURGE and only calls FIRE ?  Are the window and it's contents removed after the end time + lateness are exceeded ?

Thanks

On Monday, November 20, 2017 2:18 AM, Stefan Richter <[hidden email]> wrote:

Hi,

"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."

I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged.  Am I missing something ?  

No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.

Best,

Stefan

[Aljoscha Krettek]

Hi,

Yes, after the watermark (or processing-time) pass "end-of-window + allowed-lateness" everything that is stored for a window is deleted.

Best,

Aljoscha

On 25. Nov 2017, at 18:07, M Singh <[hidden email]> wrote:

Hi:

Another question - what happens if the trigger never calls PURGE or FIRE_AND_PURGE and only calls FIRE ?  Are the window and it's contents removed after the end time + lateness are exceeded ?

Thanks

On Monday, November 20, 2017 2:18 AM, Stefan Richter <[hidden email]> wrote:

Hi,

"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."

I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged.  Am I missing something ?  

No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.

Best,

Stefan

[M Singh]

Thanks Aljoscha.

On Sunday, November 26, 2017 11:21 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

Yes, after the watermark (or processing-time) pass "end-of-window + allowed-lateness" everything that is stored for a window is deleted.

Best,

Aljoscha

On 25. Nov 2017, at 18:07, M Singh <[hidden email]> wrote:

Hi:

Another question - what happens if the trigger never calls PURGE or FIRE_AND_PURGE and only calls FIRE ?  Are the window and it's contents removed after the end time + lateness are exceeded ?

Thanks

On Monday, November 20, 2017 2:18 AM, Stefan Richter <[hidden email]> wrote:

Hi,

"In the first case, it is a new window without the previous elements, in the second case the window reflects the old contents plus all changes since the last trigger."

I am assuming the first case is FIRE and second case is FIRE_AND_PURGE - I was thinking that in the first case (FIRE), there would be elements from previous window since we did not purge and the second case it would have only new elements since it was purged.  Am I missing something ?  

No, with first case, I meant the „truly new window case“ and the second case is „another triggering of the previous (non-purged) window“. So the second case is a simple FIRE without PURGE.

Best,

Stefan