CEP and Within Clause

[Sameer Wadkar]

Hi,

It looks like the WithIn clause of CEP uses Tumbling Windows. I could get it to use Sliding windows by using an upstream pipeline which uses Sliding Windows and produces repeating elements (in each sliding window) and applying a Watermark assigner on the resulting stream with elements duplicated. I wanted to use the "followedBy" pattern where there is a strong need for sliding windows.

Is there a plan to add sliding windows to the within clause at some point? 

The PatternStream class's "select" and "flatSelect" have overloaded versions which take PatternTimeOut variable. Is there a way to insert some of those elements back to the front of the stream. Say I am trying to find a pattern where two temperature readings >150 within 6 second window should raise an alert. If only one was found, can I insert that one back in the front of the stream on that task node (for that window pane) so that I can find a pattern match in the events occurring in the next 6 seconds. If I can do that, I don't need sliding windows. Else I cannot avoid using them for such scenarios.

Thanks,

Sameer

[Aljoscha Krettek]

+Till, looping him in directly, he probably missed this because he was away for a while.

On Tue, 26 Jul 2016 at 18:21 Sameer W <[hidden email]> wrote:

Hi,

It looks like the WithIn clause of CEP uses Tumbling Windows. I could get it to use Sliding windows by using an upstream pipeline which uses Sliding Windows and produces repeating elements (in each sliding window) and applying a Watermark assigner on the resulting stream with elements duplicated. I wanted to use the "followedBy" pattern where there is a strong need for sliding windows.

Is there a plan to add sliding windows to the within clause at some point? 

The PatternStream class's "select" and "flatSelect" have overloaded versions which take PatternTimeOut variable. Is there a way to insert some of those elements back to the front of the stream. Say I am trying to find a pattern where two temperature readings >150 within 6 second window should raise an alert. If only one was found, can I insert that one back in the front of the stream on that task node (for that window pane) so that I can find a pattern match in the events occurring in the next 6 seconds. If I can do that, I don't need sliding windows. Else I cannot avoid using them for such scenarios.

Thanks,

Sameer

[Till Rohrmann]

Hi Sameer,

the within clause of CEP uses neither tumbling nor sliding windows. It is more like a session window which is started whenever an element which matches the starting condition arrives. As long as new events which fulfill the pattern definition arrive within the length of the window, they will be added. If the pattern should not be completed within the specified time interval, the partially matched pattern will be discarded. If you've specified a timeout handler, then the timeout handler is called with the partial pattern.

At the moment, there is no way to re-insert elements in the upstream. Actually there is also no need for it because the CEP operator will detect the alert patterns if there are two temperature readings > 150 within 6 seconds.

Cheers,

Till

On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <[hidden email]> wrote:

+Till, looping him in directly, he probably missed this because he was away for a while.

On Tue, 26 Jul 2016 at 18:21 Sameer W <[hidden email]> wrote:

Hi,

It looks like the WithIn clause of CEP uses Tumbling Windows. I could get it to use Sliding windows by using an upstream pipeline which uses Sliding Windows and produces repeating elements (in each sliding window) and applying a Watermark assigner on the resulting stream with elements duplicated. I wanted to use the "followedBy" pattern where there is a strong need for sliding windows.

Is there a plan to add sliding windows to the within clause at some point? 

The PatternStream class's "select" and "flatSelect" have overloaded versions which take PatternTimeOut variable. Is there a way to insert some of those elements back to the front of the stream. Say I am trying to find a pattern where two temperature readings >150 within 6 second window should raise an alert. If only one was found, can I insert that one back in the front of the stream on that task node (for that window pane) so that I can find a pattern match in the events occurring in the next 6 seconds. If I can do that, I don't need sliding windows. Else I cannot avoid using them for such scenarios.

Thanks,

Sameer

[Sameer Wadkar]

Thanks Till,

In that case if I have a pattern -

First = T > 30

Followed By = T > 50

Within 10 minutes

If I get the following sequence of events within 10 minutes

T=31, T=51, T=31, T=51

I assume the alert will fire twice now.

But what happens if the last T=51 arrives in the 11th minute. If the partially matched pattern is discarded after 10 minutes how will the system detect T=51. Or do you mean that that timer (for the within clause) is reset each time the patter T>30 matches. In that case it would fire!

Thanks,

Sameer

On Tue, Aug 2, 2016 at 10:02 AM, Till Rohrmann <[hidden email]> wrote:

Hi Sameer,

the within clause of CEP uses neither tumbling nor sliding windows. It is more like a session window which is started whenever an element which matches the starting condition arrives. As long as new events which fulfill the pattern definition arrive within the length of the window, they will be added. If the pattern should not be completed within the specified time interval, the partially matched pattern will be discarded. If you've specified a timeout handler, then the timeout handler is called with the partial pattern.

At the moment, there is no way to re-insert elements in the upstream. Actually there is also no need for it because the CEP operator will detect the alert patterns if there are two temperature readings > 150 within 6 seconds.

Cheers,

Till

On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <[hidden email]> wrote:

+Till, looping him in directly, he probably missed this because he was away for a while.

On Tue, 26 Jul 2016 at 18:21 Sameer W <[hidden email]> wrote:

Hi,

It looks like the WithIn clause of CEP uses Tumbling Windows. I could get it to use Sliding windows by using an upstream pipeline which uses Sliding Windows and produces repeating elements (in each sliding window) and applying a Watermark assigner on the resulting stream with elements duplicated. I wanted to use the "followedBy" pattern where there is a strong need for sliding windows.

Is there a plan to add sliding windows to the within clause at some point? 

The PatternStream class's "select" and "flatSelect" have overloaded versions which take PatternTimeOut variable. Is there a way to insert some of those elements back to the front of the stream. Say I am trying to find a pattern where two temperature readings >150 within 6 second window should raise an alert. If only one was found, can I insert that one back in the front of the stream on that task node (for that window pane) so that I can find a pattern match in the events occurring in the next 6 seconds. If I can do that, I don't need sliding windows. Else I cannot avoid using them for such scenarios.

Thanks,

Sameer

[Till Rohrmann-2]

The CEP operator maintains for each pattern a window length. This means that every starting event will set its own timeout value.

So if T=51 arrives in the 11th minute, then it depends whether the second T=31 arrived sometime between the 1st and 11th minute. If that's the case, then you should also see a second matching.

Cheers,

Till

On Tue, Aug 2, 2016 at 10:20 PM, Sameer W <[hidden email]> wrote:

Thanks Till,

In that case if I have a pattern -

First = T > 30

Followed By = T > 50

Within 10 minutes

If I get the following sequence of events within 10 minutes

T=31, T=51, T=31, T=51

I assume the alert will fire twice now.

But what happens if the last T=51 arrives in the 11th minute. If the partially matched pattern is discarded after 10 minutes how will the system detect T=51. Or do you mean that that timer (for the within clause) is reset each time the patter T>30 matches. In that case it would fire!

Thanks,

Sameer

On Tue, Aug 2, 2016 at 10:02 AM, Till Rohrmann <[hidden email]> wrote:

Hi Sameer,

the within clause of CEP uses neither tumbling nor sliding windows. It is more like a session window which is started whenever an element which matches the starting condition arrives. As long as new events which fulfill the pattern definition arrive within the length of the window, they will be added. If the pattern should not be completed within the specified time interval, the partially matched pattern will be discarded. If you've specified a timeout handler, then the timeout handler is called with the partial pattern.

At the moment, there is no way to re-insert elements in the upstream. Actually there is also no need for it because the CEP operator will detect the alert patterns if there are two temperature readings > 150 within 6 seconds.

Cheers,

Till

On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <[hidden email]> wrote:

+Till, looping him in directly, he probably missed this because he was away for a while.

On Tue, 26 Jul 2016 at 18:21 Sameer W <[hidden email]> wrote:

Hi,

It looks like the WithIn clause of CEP uses Tumbling Windows. I could get it to use Sliding windows by using an upstream pipeline which uses Sliding Windows and produces repeating elements (in each sliding window) and applying a Watermark assigner on the resulting stream with elements duplicated. I wanted to use the "followedBy" pattern where there is a strong need for sliding windows.

Is there a plan to add sliding windows to the within clause at some point? 

The PatternStream class's "select" and "flatSelect" have overloaded versions which take PatternTimeOut variable. Is there a way to insert some of those elements back to the front of the stream. Say I am trying to find a pattern where two temperature readings >150 within 6 second window should raise an alert. If only one was found, can I insert that one back in the front of the stream on that task node (for that window pane) so that I can find a pattern match in the events occurring in the next 6 seconds. If I can do that, I don't need sliding windows. Else I cannot avoid using them for such scenarios.

Thanks,

Sameer

[Sameer Wadkar]

Thank you-  It is very clear now.

Sameer

On Tue, Aug 2, 2016 at 10:29 AM, Till Rohrmann <[hidden email]> wrote:

The CEP operator maintains for each pattern a window length. This means that every starting event will set its own timeout value.

So if T=51 arrives in the 11th minute, then it depends whether the second T=31 arrived sometime between the 1st and 11th minute. If that's the case, then you should also see a second matching.

Cheers,

Till

On Tue, Aug 2, 2016 at 10:20 PM, Sameer W <[hidden email]> wrote:

Thanks Till,

In that case if I have a pattern -

First = T > 30

Followed By = T > 50

Within 10 minutes

If I get the following sequence of events within 10 minutes

T=31, T=51, T=31, T=51

I assume the alert will fire twice now.

But what happens if the last T=51 arrives in the 11th minute. If the partially matched pattern is discarded after 10 minutes how will the system detect T=51. Or do you mean that that timer (for the within clause) is reset each time the patter T>30 matches. In that case it would fire!

Thanks,

Sameer

On Tue, Aug 2, 2016 at 10:02 AM, Till Rohrmann <[hidden email]> wrote:

Hi Sameer,

the within clause of CEP uses neither tumbling nor sliding windows. It is more like a session window which is started whenever an element which matches the starting condition arrives. As long as new events which fulfill the pattern definition arrive within the length of the window, they will be added. If the pattern should not be completed within the specified time interval, the partially matched pattern will be discarded. If you've specified a timeout handler, then the timeout handler is called with the partial pattern.

At the moment, there is no way to re-insert elements in the upstream. Actually there is also no need for it because the CEP operator will detect the alert patterns if there are two temperature readings > 150 within 6 seconds.

Cheers,

Till

On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <[hidden email]> wrote:

+Till, looping him in directly, he probably missed this because he was away for a while.

On Tue, 26 Jul 2016 at 18:21 Sameer W <[hidden email]> wrote:

Hi,

It looks like the WithIn clause of CEP uses Tumbling Windows. I could get it to use Sliding windows by using an upstream pipeline which uses Sliding Windows and produces repeating elements (in each sliding window) and applying a Watermark assigner on the resulting stream with elements duplicated. I wanted to use the "followedBy" pattern where there is a strong need for sliding windows.

Is there a plan to add sliding windows to the within clause at some point? 

The PatternStream class's "select" and "flatSelect" have overloaded versions which take PatternTimeOut variable. Is there a way to insert some of those elements back to the front of the stream. Say I am trying to find a pattern where two temperature readings >150 within 6 second window should raise an alert. If only one was found, can I insert that one back in the front of the stream on that task node (for that window pane) so that I can find a pattern match in the events occurring in the next 6 seconds. If I can do that, I don't need sliding windows. Else I cannot avoid using them for such scenarios.

Thanks,

Sameer