[Stateful Functions] Using Flink CEP

[Oytun Tez]

Hi there,

I was wondering if I could somehow use CEP within a Function. Have you experimented with this before?

Or, do you have any suggestions to do CEP within a Function? I am looking for a standalone library now.

Oytun

[Oytun Tez]

I am leaning towards using Siddhi as a library, but I would really love to stick with Flink CEP, or at least the specific CEP mechanism that Flink CEP uses. Exploring the codebase of Flink CEP wasn't much promising on this end.

On Mon, Apr 13, 2020 at 11:22 AM Oytun Tez <[hidden email]> wrote:

Hi there,

I was wondering if I could somehow use CEP within a Function. Have you experimented with this before?

Or, do you have any suggestions to do CEP within a Function? I am looking for a standalone library now.

Oytun

 --

Oytun Tez M O T A W O R D | CTO & Co-Founder [hidden email]

[Tzu-Li (Gordon) Tai]

Hi!

It isn't possible to use Flink CEP within Stateful Functions.

That could be an interesting primitive, to add CEP-based function
constructs.
Could your briefly describe what you are trying to achieve?

On the other hand, there are plans to integrate Stateful Functions more
closely with the Flink APIs.
One direction we've been thinking about is to, for example, support Flink
DataStreams as StateFun ingress / egresses. In this case, you'll be able to
use Flink CEP to detect patterns, and use the results as an ingress which
invokes functions within a StateFun app.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Igal Shilman]

Hi,

I'm not familiar with the other library that you have mentioned, and indeed using Flink CEP from within a stateful function is not possible within a single Flink job,  as Gordon mentioned.

I'm wondering what aspects of CEP are you interested in?

Because essentially a stateful function can be considered as a state machine with auxiliary state.

You can take a look at the ride sharing example [1] for instance, where the drivers, and the rides are cooperative state machines.

[1] - https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-ridesharing-example/statefun-ridesharing-example-functions/src/main/java/org/apache/flink/statefun/examples/ridesharing

Good luck!

Igal.

On Tue, Apr 14, 2020 at 5:07 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

It isn't possible to use Flink CEP within Stateful Functions.

That could be an interesting primitive, to add CEP-based function
constructs.
Could your briefly describe what you are trying to achieve?

On the other hand, there are plans to integrate Stateful Functions more
closely with the Flink APIs.
One direction we've been thinking about is to, for example, support Flink
DataStreams as StateFun ingress / egresses. In this case, you'll be able to
use Flink CEP to detect patterns, and use the results as an ingress which
invokes functions within a StateFun app.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Oytun Tez]

Hi Gordon,

Getting a little closer to Flink API could be helpful here with integration. DataStreams as ingress/egress would be AMAZING. Deploying regular Flink API code and statefun together as a single job is also something I will explore soon.

With CEP, I simply want to keep a Function-specific pattern engine. I could use a library like Siddhi and try to persist its state via Function's PersistValue or something. 

Also, one thing to consider in Flink API integration is protobuf enforcement in statefun. We had to use protobuf within the statefun code, but nowhere else we are using protobuf. Even if we talk via DataStreams, this would still bring us some level of separation between our Flink API code and statefun.

On Mon, Apr 13, 2020 at 11:07 PM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

It isn't possible to use Flink CEP within Stateful Functions.

That could be an interesting primitive, to add CEP-based function
constructs.
Could your briefly describe what you are trying to achieve?

On the other hand, there are plans to integrate Stateful Functions more
closely with the Flink APIs.
One direction we've been thinking about is to, for example, support Flink
DataStreams as StateFun ingress / egresses. In this case, you'll be able to
use Flink CEP to detect patterns, and use the results as an ingress which
invokes functions within a StateFun app.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Oytun Tez]

Hi Igal,

I have use cases such as "when a translator translates 10 words within 30 seconds". Typically, it is beautiful to express these with CEP.

Yet, these are exploration questions where I try to replicate our Flink application in Statefun. Rephrasing problems better might be what's needed to fit our problems within Statefun (e.g., from a CEP-like expression to the ridesharing example you shared)

Oytun

On Tue, Apr 14, 2020 at 5:41 AM Igal Shilman <[hidden email]> wrote:

Hi,

I'm not familiar with the other library that you have mentioned, and indeed using Flink CEP from within a stateful function is not possible within a single Flink job,  as Gordon mentioned.

I'm wondering what aspects of CEP are you interested in?

Because essentially a stateful function can be considered as a state machine with auxiliary state.

You can take a look at the ride sharing example [1] for instance, where the drivers, and the rides are cooperative state machines.

[1] - https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-ridesharing-example/statefun-ridesharing-example-functions/src/main/java/org/apache/flink/statefun/examples/ridesharing

Good luck!

Igal.

On Tue, Apr 14, 2020 at 5:07 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

It isn't possible to use Flink CEP within Stateful Functions.

That could be an interesting primitive, to add CEP-based function
constructs.
Could your briefly describe what you are trying to achieve?

On the other hand, there are plans to integrate Stateful Functions more
closely with the Flink APIs.
One direction we've been thinking about is to, for example, support Flink
DataStreams as StateFun ingress / egresses. In this case, you'll be able to
use Flink CEP to detect patterns, and use the results as an ingress which
invokes functions within a StateFun app.

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/