How to apply patterns from a source onto another datastream?

[Jayant Ameta]

[Dawid Wysakowicz]

It is not possible at this moment. FlinkCEP can handle only one Pattern applied statically. There is a JIRA ticket for that: https://issues.apache.org/jira/browse/FLINK-7129 .

> On 19 Dec 2017, at 10:10, Jayant Ameta <[hidden email]> wrote:
>
> I've a datastream of events, and another datastream of patterns. The patterns are provided by users at runtime, and they need to come via a Kafka topic. I need to apply each of the pattern on the event stream using Flink-CEP. Is there a way to get a PatternStream from the DataStream when I don't know the pattern beforehand?
>
> https://stackoverflow.com/questions/47883408/apache-flink-how-to-apply-patterns-from-a-source-onto-another-datastream

[Jayant Ameta]

Would it be possible to get the same result using windows?

On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz <[hidden email]> wrote:

It is not possible at this moment. FlinkCEP can handle only one Pattern applied statically. There is a JIRA ticket for that: https://issues.apache.org/jira/browse/FLINK-7129 .

> On 19 Dec 2017, at 10:10, Jayant Ameta <[hidden email]> wrote:
>
> I've a datastream of events, and another datastream of patterns. The patterns are provided by users at runtime, and they need to come via a Kafka topic. I need to apply each of the pattern on the event stream using Flink-CEP. Is there a way to get a PatternStream from the DataStream when I don't know the pattern beforehand?
>
> https://stackoverflow.com/questions/47883408/apache-flink-how-to-apply-patterns-from-a-source-onto-another-datastream

[Dawid Wysakowicz]

Hi Jayant,

Could you elaborate a bit more what you mean? Flink’s windows are not used in Flink CEP. They are a different concept.

> On 20 Dec 2017, at 09:23, Jayant Ameta <[hidden email]> wrote:
>
> Would it be possible to get the same result using windows?
>
> Jayant Ameta
>
> On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz <[hidden email]> wrote:
> It is not possible at this moment. FlinkCEP can handle only one Pattern applied statically. There is a JIRA ticket for that: https://issues.apache.org/jira/browse/FLINK-7129 .
>
> > On 19 Dec 2017, at 10:10, Jayant Ameta <[hidden email]> wrote:
> >
> > I've a datastream of events, and another datastream of patterns. The patterns are provided by users at runtime, and they need to come via a Kafka topic. I need to apply each of the pattern on the event stream using Flink-CEP. Is there a way to get a PatternStream from the DataStream when I don't know the pattern beforehand?
> >
> > https://stackoverflow.com/questions/47883408/apache-flink-how-to-apply-patterns-from-a-source-onto-another-datastream
>
>

[Jayant Ameta]

Hi Dawid,

Since dynamic patterns are not available in Flink CEP, I am thinking about skipping the CEP altogether, and mimic the functionality using windows stream. 

I am mostly interested in times and within methods. Basically, rewriting my own logic on windowed stream to match the pattern, and count the number of matching events within a time window.

Do you know if there is any similar example in the docs?

On Fri, Dec 22, 2017 at 1:24 PM, Dawid Wysakowicz <[hidden email]> wrote:

Hi Jayant,

Could you elaborate a bit more what you mean? Flink’s windows are not used in Flink CEP. They are a different concept.

> On 20 Dec 2017, at 09:23, Jayant Ameta <[hidden email]> wrote:
>
> Would it be possible to get the same result using windows?
>
> Jayant Ameta
>
> On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz <[hidden email]> wrote:
> It is not possible at this moment. FlinkCEP can handle only one Pattern applied statically. There is a JIRA ticket for that: https://issues.apache.org/jira/browse/FLINK-7129 .
>
> > On 19 Dec 2017, at 10:10, Jayant Ameta <[hidden email]> wrote:
> >
> > I've a datastream of events, and another datastream of patterns. The patterns are provided by users at runtime, and they need to come via a Kafka topic. I need to apply each of the pattern on the event stream using Flink-CEP. Is there a way to get a PatternStream from the DataStream when I don't know the pattern beforehand?
> >
> > https://stackoverflow.com/questions/47883408/apache-flink-how-to-apply-patterns-from-a-source-onto-another-datastream
>
>

[Kostas Kloudas]

Hi Jayant, 

As Dawid said, currently dynamically updating patterns is currently not supported.

There is also this question raised in the dev mailing list with 

the subject CEP: Dynamic Patterns.

I will repeat my answer here so that we are on the same page: 

"To support this, we need 2 features with one having to be added in Flink itself,

and the other to the CEP library.

The first one is broadcast state and the ability to connect keyed and non-keyed 

streams. This one is to be added to Flink itself and the good news are that this 

feature is scheduled to be added to Flink 1.5.

The second feature is to modify the CEP operator so that it can support multiple 

patterns and match incoming events against all of them. For this I have no clear 

deadline in my mind, but given that there are more and more people asking for 

it, I think it is going to be added soon."

Now for implementing CEP on top of Flink’s windowing mechanism, I would not

consider it as straight-forward as it sounds. Conceptually, CEP forms windows 

and within these windows you search for matching patterns. But CEP’s windowing 

semantics differ drastically from Flink’s windowing. Windows in CEP are created

whenever an event that matches the first element in a pattern comes. In Flink,

a window is created and is considered complete either based on time (tumbling/

sliding) or when a specific time interval expires without any activity in the stream (session).

So in one case (CEP), "window" boundaries are defined based on event properties while in the 

other (Flink Windowing), they are specified based on time. In addition, in CEP the order 

in which elements are consumed matters, as the pattern is essentially a state machine. 

In Flink, elements are added to a window in the order that they arrive.

In any case, I would consider an effort to re-implement CEP on top of Flink’s windowing

far from trivial. If your use case is simple and fits into Flink’s windowing semantics, then

go ahead. But if not, I would recommend waiting a bit more for the feature to be 

supported by the library.

Regards, 

Kostas

On Dec 25, 2017, at 5:53 AM, Jayant Ameta <[hidden email]> wrote:

Hi Dawid,

Since dynamic patterns are not available in Flink CEP, I am thinking about skipping the CEP altogether, and mimic the functionality using windows stream. 

I am mostly interested in times and within methods. Basically, rewriting my own logic on windowed stream to match the pattern, and count the number of matching events within a time window.

Do you know if there is any similar example in the docs?

On Fri, Dec 22, 2017 at 1:24 PM, Dawid Wysakowicz <[hidden email]> wrote:

Hi Jayant,

Could you elaborate a bit more what you mean? Flink’s windows are not used in Flink CEP. They are a different concept.

> On 20 Dec 2017, at 09:23, Jayant Ameta <[hidden email]> wrote:
>
> Would it be possible to get the same result using windows?
>
> Jayant Ameta
>
> On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz <[hidden email]> wrote:
> It is not possible at this moment. FlinkCEP can handle only one Pattern applied statically. There is a JIRA ticket for that: https://issues.apache.org/jira/browse/FLINK-7129 .
>
> > On 19 Dec 2017, at 10:10, Jayant Ameta <[hidden email]> wrote:
> >
> > I've a datastream of events, and another datastream of patterns. The patterns are provided by users at runtime, and they need to come via a Kafka topic. I need to apply each of the pattern on the event stream using Flink-CEP. Is there a way to get a PatternStream from the DataStream when I don't know the pattern beforehand?
> >
> > https://stackoverflow.com/questions/47883408/apache-flink-how-to-apply-patterns-from-a-source-onto-another-datastream
>
>