Flink CEP questions

[Basanth Gowda]

All,

New to Flink and more so with Flink CEP.

I want to write a sample program that does the following :

Lets suppose data cpu usage of a given server.

1. Want to Alert when CPU usage is above or below certain value
2. Want to Alert when CPU usage falls in a range
3. Want to Alert when the above condition matches n times in x interval (could be seconds, minutes, hours)
4. Want to Alert when the above condition happens continuously for x interval (could be seconds, minutes or hours) 

How would we achieve 3, 4 in the list above ? Any examples that I refer to ?

thank you,

Basanth

[Kostas Kloudas]

Hi Basanth,

This is the documentation page can be found here: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html

For:

3) you should use the times(N) and the within(TIME) clauses

  4) if by continuously you mean without stopping, then you should use the followedBy() or next() 

(check "Combining Patterns” https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html#combining-patterns in the docs above)

I am not aware of any examples but you can check this slides: 

https://www.slideshare.net/dataArtisans/kostas-kloudas-complex-event-processing-with-flink-the-state-of-flinkcep

for an overview of the CEP library or you can watch the related video.

Cheers,

Kostas

On Aug 17, 2017, at 3:32 PM, Basanth Gowda <[hidden email]> wrote:

All,

New to Flink and more so with Flink CEP.

I want to write a sample program that does the following :

Lets suppose data cpu usage of a given server.

1. Want to Alert when CPU usage is above or below certain value
2. Want to Alert when CPU usage falls in a range
3. Want to Alert when the above condition matches n times in x interval (could be seconds, minutes, hours)
4. Want to Alert when the above condition happens continuously for x interval (could be seconds, minutes or hours) 

How would we achieve 3, 4 in the list above ? Any examples that I refer to ?

thank you,

Basanth

[Basanth Gowda]

Hi Kostas,

For 3 -> I was able to do the following and it worked perfectly fine. Is there a way we could reset? Looks like the following code behaves more like a sliding count. What I want to do is reset the count once the alert has matched, and start over the count. May be I will have to have some state and do that in filter event myself ?

Pattern.<Event>begin("rule").where(new IterativeCondition<Event>() {
        @Override
        public boolean filter(Event event, Context<Event> context) throws Exception {
            event.getCpu() > 80.0;   
        }
}).times(5).within(Time.of(1, TimeUnit.MINUTES));

For 4 -> I wasn't able to do this. Assuming I don't know the frequency of Input events, I want to tell every event that came in the 1 minute, should have matched the pattern. It could be 10 events in one minute, may be 50 the next minute.  If all the events in the window match the pattern, then we want an alert. 

thank you,

Basanth

On Thu, Aug 17, 2017 at 9:46 AM, Kostas Kloudas <[hidden email]> wrote:

Hi Basanth,

This is the documentation page can be found here: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html

For:

3) you should use the times(N) and the within(TIME) clauses

  4) if by continuously you mean without stopping, then you should use the followedBy() or next() 

(check "Combining Patterns” https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html#combining-patterns in the docs above)

I am not aware of any examples but you can check this slides: 

https://www.slideshare.net/dataArtisans/kostas-kloudas-complex-event-processing-with-flink-the-state-of-flinkcep

for an overview of the CEP library or you can watch the related video.

Cheers,

Kostas

On Aug 17, 2017, at 3:32 PM, Basanth Gowda <[hidden email]> wrote:

All,

New to Flink and more so with Flink CEP.

I want to write a sample program that does the following :

Lets suppose data cpu usage of a given server.

1. Want to Alert when CPU usage is above or below certain value
2. Want to Alert when CPU usage falls in a range
3. Want to Alert when the above condition matches n times in x interval (could be seconds, minutes, hours)
4. Want to Alert when the above condition happens continuously for x interval (could be seconds, minutes or hours) 

How would we achieve 3, 4 in the list above ? Any examples that I refer to ?

thank you,

Basanth

[Biplob Biswas]

Hi Basanth,

AFAIK, CEP works like sessions window and a session is started for each event which comes in and expires at the end of the time limit.
Technically the count is kept separately for each event, so there's no reset. For ex, if you have 6 events,

1,2,3,4,5,6 (and they arrive in order like this)

then, when event 1 arrives a time counter of 1 min is started(that's your within time), same with event 2, 3,4,5,6

so, in this case, you would generate 2 alerts corresponding to first event 1 and first event 2 (as they both are followed by 4 more similar events and within the given time).

So the behaviour you are getting is expected and correct for situation 3.


For situation 4, "Want to Alert when the above condition happens continuously for x interval (could be seconds, minutes or hours) " what do you mean by continuously?

if you want an alert when you are getting multiple alerts(define multiple - what would you mean by multiple - the threshold) for situation 3,  then create a second pattern which detects the occurence of the first pattern.

a very nice example by Till can be found here, although this uses the old API, but you would get the idea:

https://github.com/tillrohrmann/cep-monitoring/blob/master/src/main/java/org/stsffap/cep/monitoring/CEPMonitoring.java

[Dawid Wysakowicz]

Hi Basanth,

Ad.3 Unfortunately right now, you cannot reset, but there is ongoing work to introduce AfterMatchSkipStrategies(https://issues.apache.org/jira/browse/FLINK-7169?filter=12339990). This will allow the behaviour you described with the SKIP_PAST_LAST strategy.

Ad.4 If I understand correctly, you would like to trigger the Pattern matching just at the end of the window. The CEP library emits the matches as soon as they are found, so I don’t think you can implement that use case with CEP. You can though try implement it yourself with e.g. ProcessFunction (https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/process_function.html)

> On 18 Aug 2017, at 03:28, Basanth Gowda <[hidden email]> wrote:
>
> Hi Kostas,
>
> For 3 -> I was able to do the following and it worked perfectly fine. Is there a way we could reset? Looks like the following code behaves more like a sliding count. What I want to do is reset the count once the alert has matched, and start over the count. May be I will have to have some state and do that in filter event myself ?
>
> Pattern.<Event>begin("rule").where(new IterativeCondition<Event>() {
>         @Override
>         public boolean filter(Event event, Context<Event> context) throws Exception {
>             event.getCpu() > 80.0;
>         }
> }).times(5).within(Time.of(1, TimeUnit.MINUTES));
>
>
> For 4 -> I wasn't able to do this. Assuming I don't know the frequency of Input events, I want to tell every event that came in the 1 minute, should have matched the pattern. It could be 10 events in one minute, may be 50 the next minute.  If all the events in the window match the pattern, then we want an alert.
>
> thank you,
> Basanth
>
> On Thu, Aug 17, 2017 at 9:46 AM, Kostas Kloudas <[hidden email]> wrote:
> Hi Basanth,
>
> This is the documentation page can be found here: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
> For:
> 3) you should use the times(N) and the within(TIME) clauses
>   4) if by continuously you mean without stopping, then you should use the followedBy() or next()
> (check "Combining Patterns” https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html#combining-patterns in the docs above)
>
> I am not aware of any examples but you can check this slides:
> https://www.slideshare.net/dataArtisans/kostas-kloudas-complex-event-processing-with-flink-the-state-of-flinkcep
> for an overview of the CEP library or you can watch the related video.
>
> Cheers,
> Kostas
>
>> On Aug 17, 2017, at 3:32 PM, Basanth Gowda <[hidden email]> wrote:
>>
>> All,
>> New to Flink and more so with Flink CEP.
>>
>> I want to write a sample program that does the following :
>>
>> Lets suppose data cpu usage of a given server.
>>
>> • Want to Alert when CPU usage is above or below certain value
>> • Want to Alert when CPU usage falls in a range
>> • Want to Alert when the above condition matches n times in x interval (could be seconds, minutes, hours)
>> • Want to Alert when the above condition happens continuously for x interval (could be seconds, minutes or hours)
>> How would we achieve 3, 4 in the list above ? Any examples that I refer to ?
>>
>>
>> thank you,
>> Basanth
>
>

[Basanth Gowda]

Thank you very much Biplob and David

Thanks David for those links . That is exactly what I was looking for.

On Fri, Aug 18, 2017 at 5:16 AM, Dawid Wysakowicz <[hidden email]> wrote:

Hi Basanth,

Ad.3 Unfortunately right now, you cannot reset, but there is ongoing work to introduce AfterMatchSkipStrategies(https://issues.apache.org/jira/browse/FLINK-7169?filter=12339990). This will allow the behaviour you described with the SKIP_PAST_LAST strategy.

Ad.4 If I understand correctly, you would like to trigger the Pattern matching just at the end of the window. The CEP library emits the matches as soon as they are found, so I don’t think you can implement that use case with CEP. You can though try implement it yourself with e.g. ProcessFunction (https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/process_function.html)

> On 18 Aug 2017, at 03:28, Basanth Gowda <[hidden email]> wrote:
>
> Hi Kostas,
>
> For 3 -> I was able to do the following and it worked perfectly fine. Is there a way we could reset? Looks like the following code behaves more like a sliding count. What I want to do is reset the count once the alert has matched, and start over the count. May be I will have to have some state and do that in filter event myself ?
>
> Pattern.<Event>begin("rule").where(new IterativeCondition<Event>() {
>         @Override
>         public boolean filter(Event event, Context<Event> context) throws Exception {
>             event.getCpu() > 80.0;
>         }
> }).times(5).within(Time.of(1, TimeUnit.MINUTES));
>
>
> For 4 -> I wasn't able to do this. Assuming I don't know the frequency of Input events, I want to tell every event that came in the 1 minute, should have matched the pattern. It could be 10 events in one minute, may be 50 the next minute.  If all the events in the window match the pattern, then we want an alert.
>
> thank you,
> Basanth
>
> On Thu, Aug 17, 2017 at 9:46 AM, Kostas Kloudas <[hidden email]> wrote:
> Hi Basanth,
>
> This is the documentation page can be found here: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
> For:
>       3) you should use the times(N) and the within(TIME) clauses
>       4) if by continuously you mean without stopping, then you should use the followedBy() or next()
>               (check "Combining Patterns” https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html#combining-patterns in the docs above)
>
> I am not aware of any examples but you can check this slides:
> https://www.slideshare.net/dataArtisans/kostas-kloudas-complex-event-processing-with-flink-the-state-of-flinkcep
> for an overview of the CEP library or you can watch the related video.
>
> Cheers,
> Kostas
>
>> On Aug 17, 2017, at 3:32 PM, Basanth Gowda <[hidden email]> wrote:
>>
>> All,
>> New to Flink and more so with Flink CEP.
>>
>> I want to write a sample program that does the following :
>>
>> Lets suppose data cpu usage of a given server.
>>
>>      • Want to Alert when CPU usage is above or below certain value
>>      • Want to Alert when CPU usage falls in a range
>>      • Want to Alert when the above condition matches n times in x interval (could be seconds, minutes, hours)
>>      • Want to Alert when the above condition happens continuously for x interval (could be seconds, minutes or hours)
>> How would we achieve 3, 4 in the list above ? Any examples that I refer to ?
>>
>>
>> thank you,
>> Basanth
>
>