Apache siddhi into Flink

[Aparup Banerjee (apbanerj)]

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Chen Qin]

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Trevor Grant]

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Hao Chen]

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Trevor Grant]

Thank you for confirming Hao,

Aparup, please don't refer to it as "Apache Siddhi", that is misleading. 

On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <[hidden email]> wrote:

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Aparup Banerjee (apbanerj)]

Sorry for the semantic difference. 

On Aug 28, 2016, at 12:05 PM, Trevor Grant <[hidden email]> wrote:

Thank you for confirming Hao,

Aparup, please don't refer to it as "Apache Siddhi", that is misleading. 

On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <[hidden email]> wrote:

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Chesnay Schepler]

Hello Aparup,

could you provide more information about Siddhi? How mature is it; how is the community? How does it compare to the Flink's CEP library?

How should this integration look like? Are you proposing to replace the current CEP library, or will they co-exist with different use-cases for each?

If we used Siddhi in Flink, how exactly would Flink's runtime be involved in the processing?

Regards,
Chesnay

On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote:

Sorry for the semantic difference. 

On Aug 28, 2016, at 12:05 PM, Trevor Grant <[hidden email]> wrote:

Thank you for confirming Hao,

Aparup, please don't refer to it as "Apache Siddhi", that is misleading. 

On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <[hidden email]> wrote:

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Aparup Banerjee (apbanerj)]

I think siddhi is a fairly matured CEP library. I am thinking it should co-exist with existing CEP library. My thinking is we should be able to use Siddhi QL/ Siddhi Patterns on top of flink data streams. This can co-exist naturally with existing Java / Scala based Flink CEP Library. I am still reading up on Flink internals – but at high level I am thinking about a new Flink operator on DataStream for this. Thoughts ?

Thanks,

Aparup

From: Chesnay Schepler <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, August 29, 2016 at 12:35 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Apache siddhi into Flink

Hello Aparup,

could you provide more information about Siddhi? How mature is it; how is the community? How does it compare to the Flink's CEP library?

How should this integration look like? Are you proposing to replace the current CEP library, or will they co-exist with different use-cases for each?

If we used Siddhi in Flink, how exactly would Flink's runtime be involved in the processing?

Regards,
Chesnay

On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote:

Sorry for the semantic difference. 

On Aug 28, 2016, at 12:05 PM, Trevor Grant <[hidden email][hidden email]> wrote:

Thank you for confirming Hao,

Aparup, please don't refer to it as "Apache Siddhi", that is misleading. 

On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <[hidden email]> wrote:

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email][hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Stephan Ewen]

Nice idea!

If you look at the current CEP library, it is simply a custom operator.

Often, you can even get away with a custom FlatMapFunction that uses state: https://ci.apache.org/projects/flink/flink-docs-master/dev/state.html#using-the-keyvalue-state-interface

Stephan

On Mon, Aug 29, 2016 at 10:09 AM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

I think siddhi is a fairly matured CEP library. I am thinking it should co-exist with existing CEP library. My thinking is we should be able to use Siddhi QL/ Siddhi Patterns on top of flink data streams. This can co-exist naturally with existing Java / Scala based Flink CEP Library. I am still reading up on Flink internals – but at high level I am thinking about a new Flink operator on DataStream for this. Thoughts ?

Thanks,

Aparup

From: Chesnay Schepler <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, August 29, 2016 at 12:35 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Apache siddhi into Flink

Hello Aparup,

could you provide more information about Siddhi? How mature is it; how is the community? How does it compare to the Flink's CEP library?

How should this integration look like? Are you proposing to replace the current CEP library, or will they co-exist with different use-cases for each?

If we used Siddhi in Flink, how exactly would Flink's runtime be involved in the processing?

Regards,
Chesnay

On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote:

Sorry for the semantic difference. 

On Aug 28, 2016, at 12:05 PM, Trevor Grant <[hidden email][hidden email]> wrote:

Thank you for confirming Hao,

Aparup, please don't refer to it as "Apache Siddhi", that is misleading. 

On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <[hidden email]> wrote:

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email][hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Till Rohrmann]

Hi Aparup,

I haven't looked in detail at Siddhi's internals especially the way it handles distributed execution. I've only seen that it uses Hazelcast for a distributed in-memory cache. If a distributed cache for the communication between instances of the CEP operator is needed, then you would have to integrate that as well. Otherwise, if the state stored in the operators is disjunct, then you should be good to go with Flink's state abstraction.

If you want to work on the integration, then it's best to open a JIRA issue and describe a little bit the way the integration could look like.

Cheers,

Till

On Mon, Aug 29, 2016 at 10:59 AM, Stephan Ewen <[hidden email]> wrote:

Nice idea!

If you look at the current CEP library, it is simply a custom operator.

Often, you can even get away with a custom FlatMapFunction that uses state: https://ci.apache.org/projects/flink/flink-docs-master/dev/state.html#using-the-keyvalue-state-interface

Stephan

On Mon, Aug 29, 2016 at 10:09 AM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

I think siddhi is a fairly matured CEP library. I am thinking it should co-exist with existing CEP library. My thinking is we should be able to use Siddhi QL/ Siddhi Patterns on top of flink data streams. This can co-exist naturally with existing Java / Scala based Flink CEP Library. I am still reading up on Flink internals – but at high level I am thinking about a new Flink operator on DataStream for this. Thoughts ?

Thanks,

Aparup

From: Chesnay Schepler <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, August 29, 2016 at 12:35 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Apache siddhi into Flink

Hello Aparup,

could you provide more information about Siddhi? How mature is it; how is the community? How does it compare to the Flink's CEP library?

How should this integration look like? Are you proposing to replace the current CEP library, or will they co-exist with different use-cases for each?

If we used Siddhi in Flink, how exactly would Flink's runtime be involved in the processing?

Regards,
Chesnay

On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote:

Sorry for the semantic difference. 

On Aug 28, 2016, at 12:05 PM, Trevor Grant <[hidden email][hidden email]> wrote:

Thank you for confirming Hao,

Aparup, please don't refer to it as "Apache Siddhi", that is misleading. 

On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <[hidden email]> wrote:

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email][hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup

[Hao Chen]

+1 support siddhi as a flink operator on DataStream, siddhi support rich CEP features other than pattern match, also support extensible snapshot/restore interface for fault-tolerance which should be easy to integrate with flink's state management (tested with some draft code), we (apache eagle community) use siddhi a lot to integrate with Storm and Spark for real-time monitoring cases. And siddhi's community is active as well in releasing, bug fixing, and accepting external contribution etc.

On Mon, Aug 29, 2016 at 5:21 PM, Till Rohrmann <[hidden email]> wrote:

Hi Aparup,

I haven't looked in detail at Siddhi's internals especially the way it handles distributed execution. I've only seen that it uses Hazelcast for a distributed in-memory cache. If a distributed cache for the communication between instances of the CEP operator is needed, then you would have to integrate that as well. Otherwise, if the state stored in the operators is disjunct, then you should be good to go with Flink's state abstraction.

If you want to work on the integration, then it's best to open a JIRA issue and describe a little bit the way the integration could look like.

Cheers,

Till

On Mon, Aug 29, 2016 at 10:59 AM, Stephan Ewen <[hidden email]> wrote:

Nice idea!

If you look at the current CEP library, it is simply a custom operator.

Often, you can even get away with a custom FlatMapFunction that uses state: https://ci.apache.org/projects/flink/flink-docs-master/dev/state.html#using-the-keyvalue-state-interface

Stephan

On Mon, Aug 29, 2016 at 10:09 AM, Aparup Banerjee (apbanerj) <[hidden email]> wrote:

I think siddhi is a fairly matured CEP library. I am thinking it should co-exist with existing CEP library. My thinking is we should be able to use Siddhi QL/ Siddhi Patterns on top of flink data streams. This can co-exist naturally with existing Java / Scala based Flink CEP Library. I am still reading up on Flink internals – but at high level I am thinking about a new Flink operator on DataStream for this. Thoughts ?

Thanks,

Aparup

From: Chesnay Schepler <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, August 29, 2016 at 12:35 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Apache siddhi into Flink

Hello Aparup,

could you provide more information about Siddhi? How mature is it; how is the community? How does it compare to the Flink's CEP library?

How should this integration look like? Are you proposing to replace the current CEP library, or will they co-exist with different use-cases for each?

If we used Siddhi in Flink, how exactly would Flink's runtime be involved in the processing?

Regards,
Chesnay

On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote:

Sorry for the semantic difference. 

On Aug 28, 2016, at 12:05 PM, Trevor Grant <[hidden email][hidden email]> wrote:

Thank you for confirming Hao,

Aparup, please don't refer to it as "Apache Siddhi", that is misleading. 

On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <[hidden email]> wrote:

Siddhi is not apache project, but licensed under apache license v2, being open sourced and maintained by wso2.

On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <[hidden email]> wrote:

Aparup,

Was Siddhi recently added as an incubator project?  I can't find it in the project directory or or on github.com/apache.  The closest thing I can find is this: https://github.com/wso2/siddhi

tg

On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <[hidden email]> wrote:

​+1​

On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <[hidden email][hidden email]> wrote:

Hi-

Has anyone looked into embedding apache siddhi into Flink. 

Thanks,

Aparup