Integrating Flink CEP with a Rules Engine

[Sridhar Chellappa]

Folks,

I am new to Flink.

One of the reasons why I am interested in Flink is because of its CEP library. Our CEP logic comprises of a set of complex business rules which will have to be managed (Create, Update, Delete) by a bunch of business analysts.

Is there a way I can integrate other third party tools (Drools, OpenRules) to let Business Analysts define rules and  execute them using Flink's CEP library?

[Jörn Franke]

Hallo,

It si possible, but some caveat : flink is a distributed system, but in drools the fact are only locally available. This may lead to strange effects when rules update the fact base.

Best regards

> On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP library. Our CEP logic comprises of a set of complex business rules which will have to be managed (Create, Update, Delete) by a bunch of business analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRules) to let Business Analysts define rules and  execute them using Flink's CEP library?

[Kostas Kloudas]

Hi Jorn and Sridhar,

It would be worth describing a bit more what these tools are and what are your needs.

In addition, and to see what the CEP library already offers here you can find the documentation:

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html

Thanks,

Kostas

On Jun 23, 2017, at 1:41 PM, Jörn Franke <[hidden email]> wrote:

Hallo,

It si possible, but some caveat : flink is a distributed system, but in drools the fact are only locally available. This may lead to strange effects when rules update the fact base.

Best regards

On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:

Folks,

I am new to Flink.

One of the reasons why I am interested in Flink is because of its CEP library. Our CEP logic comprises of a set of complex business rules which will have to be managed (Create, Update, Delete) by a bunch of business analysts.

Is there a way I can integrate other third party tools (Drools, OpenRules) to let Business Analysts define rules and  execute them using Flink's CEP library?

[Ismaël Mejía]

Hello,

It is really interesting to see this discussion because that was one
of the questions on the presentation on CEP at Berlin Buzzwords, and
this is one line of work that may eventually make sense to explore.

Rule engines like drools implement the Rete algorithm that if I
understood correctly optimizes the analysis of a relatively big set of
facts (conditions) into a simpler evaluation graph. For more details
this is a really nice explanation.
https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/

On flink's CEP I have the impression that you define this graph by
hand. Using a rule engine you could infer an optimal graph from the
set of rules, and then this graph could be translated into CEP
patterns.

Of course take all of this with a grain of salt because I am not an
expert on both CEP or the Rete algorithm, but I start to see the
connection of both worlds more clearly now. So if anyone else has
ideas of the feasibility of this or can see some other
issues/consequences please comment. I also have the impression that
distribution is less of an issue because the rete network is
calculated only once and updates are not 'dynamic' (but I might be
wrong).

Ismaël

ps. I add Thomas in copy who was who made the question in the
conference in case he has some comments/ideas.


On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
<[hidden email]> wrote:

> Hi Jorn and Sridhar,
>
> It would be worth describing a bit more what these tools are and what are
> your needs.
> In addition, and to see what the CEP library already offers here you can
> find the documentation:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
>
>
> Thanks,
> Kostas
>
> On Jun 23, 2017, at 1:41 PM, Jörn Franke <[hidden email]> wrote:
>
> Hallo,
>
> It si possible, but some caveat : flink is a distributed system, but in
> drools the fact are only locally available. This may lead to strange effects
> when rules update the fact base.
>
> Best regards
>
> On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP
> library. Our CEP logic comprises of a set of complex business rules which
> will have to be managed (Create, Update, Delete) by a bunch of business
> analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRules)
> to let Business Analysts define rules and  execute them using Flink's CEP
> library?
>
>

[Suneel Marthi]

FWIW, here's an old Cloudera blog about using Drools with Spark.

https://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-processing-app-on-apache-spark-and-drools/

It should be possible to invoke Drools from Flink in a similar way (I have not tried it). 

It all depends on what the use case and how much of present Flink CEP satisfies the use case before considering integration with more complex rule engines.

Disclaimer: I work for Red Hat

On Fri, Jun 23, 2017 at 8:43 AM, Ismaël Mejía <[hidden email]> wrote:

Hello,

It is really interesting to see this discussion because that was one
of the questions on the presentation on CEP at Berlin Buzzwords, and
this is one line of work that may eventually make sense to explore.

Rule engines like drools implement the Rete algorithm that if I
understood correctly optimizes the analysis of a relatively big set of
facts (conditions) into a simpler evaluation graph. For more details
this is a really nice explanation.
https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/

On flink's CEP I have the impression that you define this graph by
hand. Using a rule engine you could infer an optimal graph from the
set of rules, and then this graph could be translated into CEP
patterns.

Of course take all of this with a grain of salt because I am not an
expert on both CEP or the Rete algorithm, but I start to see the
connection of both worlds more clearly now. So if anyone else has
ideas of the feasibility of this or can see some other
issues/consequences please comment. I also have the impression that
distribution is less of an issue because the rete network is
calculated only once and updates are not 'dynamic' (but I might be
wrong).

Ismaël

ps. I add Thomas in copy who was who made the question in the
conference in case he has some comments/ideas.

On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
<[hidden email]> wrote:
> Hi Jorn and Sridhar,
>
> It would be worth describing a bit more what these tools are and what are
> your needs.
> In addition, and to see what the CEP library already offers here you can
> find the documentation:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
>
>
> Thanks,
> Kostas
>
> On Jun 23, 2017, at 1:41 PM, Jörn Franke <[hidden email]> wrote:
>
> Hallo,
>
> It si possible, but some caveat : flink is a distributed system, but in
> drools the fact are only locally available. This may lead to strange effects
> when rules update the fact base.
>
> Best regards
>
> On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP
> library. Our CEP logic comprises of a set of complex business rules which
> will have to be managed (Create, Update, Delete) by a bunch of business
> analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRules)
> to let Business Analysts define rules and  execute them using Flink's CEP
> library?
>
>

[Kostas Kloudas]

Hi all,

Currently there is an ongoing effort to integrate FlinkCEP with Flink's SQL API.

There is already an open FLIP for this:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-20%3A+Integration+of+SQL+and+CEP

So, if there was an effort for integration of different libraries/tools/functionality as well, it 

would be nice to go a bit more into details on i) what is already there, ii) what is planned to be 

integrated for the SQL effort, and iii) what else is required, and consolidate the resources

available.

This will allow the community to move faster and with a clear roadmap.

Kostas

On Jun 23, 2017, at 2:51 PM, Suneel Marthi <[hidden email]> wrote:

FWIW, here's an old Cloudera blog about using Drools with Spark.

https://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-processing-app-on-apache-spark-and-drools/

It should be possible to invoke Drools from Flink in a similar way (I have not tried it). 

It all depends on what the use case and how much of present Flink CEP satisfies the use case before considering integration with more complex rule engines.

Disclaimer: I work for Red Hat

On Fri, Jun 23, 2017 at 8:43 AM, Ismaël Mejía <[hidden email]> wrote:

Hello,

It is really interesting to see this discussion because that was one
of the questions on the presentation on CEP at Berlin Buzzwords, and
this is one line of work that may eventually make sense to explore.

Rule engines like drools implement the Rete algorithm that if I
understood correctly optimizes the analysis of a relatively big set of
facts (conditions) into a simpler evaluation graph. For more details
this is a really nice explanation.
https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/

On flink's CEP I have the impression that you define this graph by
hand. Using a rule engine you could infer an optimal graph from the
set of rules, and then this graph could be translated into CEP
patterns.

Of course take all of this with a grain of salt because I am not an
expert on both CEP or the Rete algorithm, but I start to see the
connection of both worlds more clearly now. So if anyone else has
ideas of the feasibility of this or can see some other
issues/consequences please comment. I also have the impression that
distribution is less of an issue because the rete network is
calculated only once and updates are not 'dynamic' (but I might be
wrong).

Ismaël

ps. I add Thomas in copy who was who made the question in the
conference in case he has some comments/ideas.

On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
<[hidden email]> wrote:
> Hi Jorn and Sridhar,
>
> It would be worth describing a bit more what these tools are and what are
> your needs.
> In addition, and to see what the CEP library already offers here you can
> find the documentation:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
>
>
> Thanks,
> Kostas
>
> On Jun 23, 2017, at 1:41 PM, Jörn Franke <[hidden email]> wrote:
>
> Hallo,
>
> It si possible, but some caveat : flink is a distributed system, but in
> drools the fact are only locally available. This may lead to strange effects
> when rules update the fact base.
>
> Best regards
>
> On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP
> library. Our CEP logic comprises of a set of complex business rules which
> will have to be managed (Create, Update, Delete) by a bunch of business
> analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRules)
> to let Business Analysts define rules and  execute them using Flink's CEP
> library?
>
>

[Sridhar Chellappa]

Folks,

Plenty of very good points but I see this discussion digressing from what I originally asked for. We need a dashboard to let the Business Analysts to define rules and the CEP to run them.

My original question was how to solve this with Flink CEP?

From what I see, this is not a solved problem. Correct me if I am wrong.

On Fri, Jun 23, 2017 at 6:52 PM, Kostas Kloudas <[hidden email]> wrote:

Hi all,

Currently there is an ongoing effort to integrate FlinkCEP with Flink's SQL API.

There is already an open FLIP for this:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-20%3A+Integration+of+SQL+and+CEP

So, if there was an effort for integration of different libraries/tools/functionality as well, it 

would be nice to go a bit more into details on i) what is already there, ii) what is planned to be 

integrated for the SQL effort, and iii) what else is required, and consolidate the resources

available.

This will allow the community to move faster and with a clear roadmap.

Kostas

On Jun 23, 2017, at 2:51 PM, Suneel Marthi <[hidden email]> wrote:

FWIW, here's an old Cloudera blog about using Drools with Spark.

https://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-processing-app-on-apache-spark-and-drools/

It should be possible to invoke Drools from Flink in a similar way (I have not tried it). 

It all depends on what the use case and how much of present Flink CEP satisfies the use case before considering integration with more complex rule engines.

Disclaimer: I work for Red Hat

On Fri, Jun 23, 2017 at 8:43 AM, Ismaël Mejía <[hidden email]> wrote:

Hello,

It is really interesting to see this discussion because that was one
of the questions on the presentation on CEP at Berlin Buzzwords, and
this is one line of work that may eventually make sense to explore.

Rule engines like drools implement the Rete algorithm that if I
understood correctly optimizes the analysis of a relatively big set of
facts (conditions) into a simpler evaluation graph. For more details
this is a really nice explanation.
https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/

On flink's CEP I have the impression that you define this graph by
hand. Using a rule engine you could infer an optimal graph from the
set of rules, and then this graph could be translated into CEP
patterns.

Of course take all of this with a grain of salt because I am not an
expert on both CEP or the Rete algorithm, but I start to see the
connection of both worlds more clearly now. So if anyone else has
ideas of the feasibility of this or can see some other
issues/consequences please comment. I also have the impression that
distribution is less of an issue because the rete network is
calculated only once and updates are not 'dynamic' (but I might be
wrong).

Ismaël

ps. I add Thomas in copy who was who made the question in the
conference in case he has some comments/ideas.

On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
<[hidden email]> wrote:
> Hi Jorn and Sridhar,
>
> It would be worth describing a bit more what these tools are and what are
> your needs.
> In addition, and to see what the CEP library already offers here you can
> find the documentation:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
>
>
> Thanks,
> Kostas
>
> On Jun 23, 2017, at 1:41 PM, Jörn Franke <[hidden email]> wrote:
>
> Hallo,
>
> It si possible, but some caveat : flink is a distributed system, but in
> drools the fact are only locally available. This may lead to strange effects
> when rules update the fact base.
>
> Best regards
>
> On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP
> library. Our CEP logic comprises of a set of complex business rules which
> will have to be managed (Create, Update, Delete) by a bunch of business
> analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRules)
> to let Business Analysts define rules and  execute them using Flink's CEP
> library?
>
>

[Kostas Kloudas]

The rules, or patterns supported by FlinkCEP are presented in the documentation link

I posted earlier.

Dynamically updating these patterns, is not supported yet, but there are discussions to 

add this feature soon.

If the rules you need are supported by the current version of FlinkCEP, then you can 

start right away. If not, you need to provide more details.

Kostas

On Jun 23, 2017, at 3:54 PM, Sridhar Chellappa <[hidden email]> wrote:

CEP

[Suneel Marthi]

Sorry I didn't read the whole thread. 

We have a similar rqmt wherein the users would like to add/update/delete CEP patterns via UX or REST api and we started discussing building a REST api for that, glad to see that this is a common ask and if there's already a community effort around this - that's great to know. 

On Fri, Jun 23, 2017 at 9:54 AM, Sridhar Chellappa <[hidden email]> wrote:

Folks,

Plenty of very good points but I see this discussion digressing from what I originally asked for. We need a dashboard to let the Business Analysts to define rules and the CEP to run them.

My original question was how to solve this with Flink CEP?

From what I see, this is not a solved problem. Correct me if I am wrong.

On Fri, Jun 23, 2017 at 6:52 PM, Kostas Kloudas <[hidden email]> wrote:

Hi all,

Currently there is an ongoing effort to integrate FlinkCEP with Flink's SQL API.

There is already an open FLIP for this:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-20%3A+Integration+of+SQL+and+CEP

So, if there was an effort for integration of different libraries/tools/functionality as well, it 

would be nice to go a bit more into details on i) what is already there, ii) what is planned to be 

integrated for the SQL effort, and iii) what else is required, and consolidate the resources

available.

This will allow the community to move faster and with a clear roadmap.

Kostas

On Jun 23, 2017, at 2:51 PM, Suneel Marthi <[hidden email]> wrote:

FWIW, here's an old Cloudera blog about using Drools with Spark.

https://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-processing-app-on-apache-spark-and-drools/

It should be possible to invoke Drools from Flink in a similar way (I have not tried it). 

It all depends on what the use case and how much of present Flink CEP satisfies the use case before considering integration with more complex rule engines.

Disclaimer: I work for Red Hat

On Fri, Jun 23, 2017 at 8:43 AM, Ismaël Mejía <[hidden email]> wrote:

Hello,

It is really interesting to see this discussion because that was one
of the questions on the presentation on CEP at Berlin Buzzwords, and
this is one line of work that may eventually make sense to explore.

Rule engines like drools implement the Rete algorithm that if I
understood correctly optimizes the analysis of a relatively big set of
facts (conditions) into a simpler evaluation graph. For more details
this is a really nice explanation.
https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/

On flink's CEP I have the impression that you define this graph by
hand. Using a rule engine you could infer an optimal graph from the
set of rules, and then this graph could be translated into CEP
patterns.

Of course take all of this with a grain of salt because I am not an
expert on both CEP or the Rete algorithm, but I start to see the
connection of both worlds more clearly now. So if anyone else has
ideas of the feasibility of this or can see some other
issues/consequences please comment. I also have the impression that
distribution is less of an issue because the rete network is
calculated only once and updates are not 'dynamic' (but I might be
wrong).

Ismaël

ps. I add Thomas in copy who was who made the question in the
conference in case he has some comments/ideas.

On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
<[hidden email]> wrote:
> Hi Jorn and Sridhar,
>
> It would be worth describing a bit more what these tools are and what are
> your needs.
> In addition, and to see what the CEP library already offers here you can
> find the documentation:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
>
>
> Thanks,
> Kostas
>
> On Jun 23, 2017, at 1:41 PM, Jörn Franke <[hidden email]> wrote:
>
> Hallo,
>
> It si possible, but some caveat : flink is a distributed system, but in
> drools the fact are only locally available. This may lead to strange effects
> when rules update the fact base.
>
> Best regards
>
> On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP
> library. Our CEP logic comprises of a set of complex business rules which
> will have to be managed (Create, Update, Delete) by a bunch of business
> analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRules)
> to let Business Analysts define rules and  execute them using Flink's CEP
> library?
>
>

[Anton]

We also have a requirement of using Drools in Flink. Drools brings a very mature and usable business rules editor. And to be able to integrate Drools into Flink would be very useful.

On 23 June 2017 at 22:09, Suneel Marthi <[hidden email]> wrote:

Sorry I didn't read the whole thread. 

We have a similar rqmt wherein the users would like to add/update/delete CEP patterns via UX or REST api and we started discussing building a REST api for that, glad to see that this is a common ask and if there's already a community effort around this - that's great to know. 

On Fri, Jun 23, 2017 at 9:54 AM, Sridhar Chellappa <[hidden email]> wrote:

Folks,

Plenty of very good points but I see this discussion digressing from what I originally asked for. We need a dashboard to let the Business Analysts to define rules and the CEP to run them.

My original question was how to solve this with Flink CEP?

From what I see, this is not a solved problem. Correct me if I am wrong.

On Fri, Jun 23, 2017 at 6:52 PM, Kostas Kloudas <[hidden email]> wrote:

Hi all,

Currently there is an ongoing effort to integrate FlinkCEP with Flink's SQL API.

There is already an open FLIP for this:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-20%3A+Integration+of+SQL+and+CEP

So, if there was an effort for integration of different libraries/tools/functionality as well, it 

would be nice to go a bit more into details on i) what is already there, ii) what is planned to be 

integrated for the SQL effort, and iii) what else is required, and consolidate the resources

available.

This will allow the community to move faster and with a clear roadmap.

Kostas

On Jun 23, 2017, at 2:51 PM, Suneel Marthi <[hidden email]> wrote:

FWIW, here's an old Cloudera blog about using Drools with Spark.

https://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-processing-app-on-apache-spark-and-drools/

It should be possible to invoke Drools from Flink in a similar way (I have not tried it). 

It all depends on what the use case and how much of present Flink CEP satisfies the use case before considering integration with more complex rule engines.

Disclaimer: I work for Red Hat

On Fri, Jun 23, 2017 at 8:43 AM, Ismaël Mejía <[hidden email]> wrote:

Hello,

It is really interesting to see this discussion because that was one
of the questions on the presentation on CEP at Berlin Buzzwords, and
this is one line of work that may eventually make sense to explore.

Rule engines like drools implement the Rete algorithm that if I
understood correctly optimizes the analysis of a relatively big set of
facts (conditions) into a simpler evaluation graph. For more details
this is a really nice explanation.
https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/

On flink's CEP I have the impression that you define this graph by
hand. Using a rule engine you could infer an optimal graph from the
set of rules, and then this graph could be translated into CEP
patterns.

Of course take all of this with a grain of salt because I am not an
expert on both CEP or the Rete algorithm, but I start to see the
connection of both worlds more clearly now. So if anyone else has
ideas of the feasibility of this or can see some other
issues/consequences please comment. I also have the impression that
distribution is less of an issue because the rete network is
calculated only once and updates are not 'dynamic' (but I might be
wrong).

Ismaël

ps. I add Thomas in copy who was who made the question in the
conference in case he has some comments/ideas.

On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
<[hidden email]> wrote:
> Hi Jorn and Sridhar,
>
> It would be worth describing a bit more what these tools are and what are
> your needs.
> In addition, and to see what the CEP library already offers here you can
> find the documentation:
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
>
>
> Thanks,
> Kostas
>
> On Jun 23, 2017, at 1:41 PM, Jörn Franke <[hidden email]> wrote:
>
> Hallo,
>
> It si possible, but some caveat : flink is a distributed system, but in
> drools the fact are only locally available. This may lead to strange effects
> when rules update the fact base.
>
> Best regards
>
> On 23. Jun 2017, at 12:49, Sridhar Chellappa <[hidden email]> wrote:
>
> Folks,
>
> I am new to Flink.
>
> One of the reasons why I am interested in Flink is because of its CEP
> library. Our CEP logic comprises of a set of complex business rules which
> will have to be managed (Create, Update, Delete) by a bunch of business
> analysts.
>
> Is there a way I can integrate other third party tools (Drools, OpenRules)
> to let Business Analysts define rules and  execute them using Flink's CEP
> library?
>
>