Re: [PROPOSAL] Contribute Stateful Functions to Apache Flink

Posted by Dawid Wysakowicz-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/PROPOSAL-Contribute-Stateful-Functions-to-Apache-Flink-tp30468p30476.html

Hi Stephan,

I think this is a nice library, but what I like more about it is that it suggests exploring different use-cases. I think it definitely makes sense for the Flink community to explore more lightweight applications that reuses resources. Therefore I definitely think it is a good idea for Flink community to accept this contribution and help maintaining it.

Personally I'd prefer to have it in a separate repository. There were a few discussions before where different people were suggesting to extract connectors and other libraries to separate repositories. Moreover I think it could serve as an example for the Flink ecosystem website[1]. This could be the first project in there and give a good impression that the community sees potential in the ecosystem website.

Lastly, I'm wondering if this should go through PMC vote according to our bylaws[2]. In the end the suggestion is to adopt an existing code base as is. It also proposes a new programs concept that could result in a shift of priorities for the community in a long run.

Best,

Dawid

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Create-a-Flink-ecosystem-website-td27519.html

[2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws

On 11/10/2019 13:12, Till Rohrmann wrote:

Hi Stephan,

+1 for adding stateful functions to Flink. I believe the new set of applications this feature will unlock will be super interesting for new and existing Flink users alike.

One reason for not including it in the main repository would to not being bound to Flink's release cadence. This would allow to release faster and more often. However, I believe that having it eventually in Flink's main repository would be beneficial in the long run.

Cheers,

Till

On Fri, Oct 11, 2019 at 12:56 PM Trevor Grant <[hidden email]> wrote:

+1 non-binding on contribution. 

Separate repo, or feature branch to start maybe? I just feel like in the beginning this thing is going to have lots of breaking changes that maybe aren't going to fit well with tests / other "v1+" release code. Just my .02. 

On Fri, Oct 11, 2019 at 4:38 AM Stephan Ewen <[hidden email]> wrote:

Dear Flink Community!

Some of you probably heard it already: On Tuesday, at Flink Forward Berlin, we announced **Stateful Functions**.

Stateful Functions is a library on Flink to implement general purpose applications. It is built around stateful functions (who would have thunk)

that can communicate arbitrarily through messages, have consistent state, and a small resource footprint. They are a bit like keyed ProcessFunctions

that can send each other messages.

As simple as this sounds, this means you can now communicate in non-DAG patterns, so it allows users to build programs they cannot build with Flink.

It also has other neat properties, like multiplexing of functions, modular composition, tooling both container-based deployments and as-a-Flink-job deployments.

You can find out more about it here

  - Website: https://statefun.io/

  - Code: https://github.com/ververica/stateful-functions

  - Talk with motivation: https://speakerdeck.com/stephanewen/stateful-functions-building-general-purpose-applications-and-services-on-apache-flink?slide=12

Now for the main issue: **We would like to contribute this project to Apache Flink**

I believe that this is a great fit for both sides.

For the Flink community, it would be a way to extend the capabilities and use cases of Flink into a completely different type of applications and thus grow the community into this new field.

Many discussions recently about evolving the Flink runtime (both on the mailing list and at conferences) show the interest in Flink users in the space that Stateful Functions covers.

It seems natural that Stateful Functions should closely co-develop with Apache Flink, ideally as part of the project.

There are many details to be discusses, for example whether this should be added to the Flink core repository, or whether we and to create a separate repository

for this. But I think we should start discussing this after we have consensus on whether the community wants this contribution.

Really looking forward to hear what you think!

Best Regards,

Stephan