Flink Stateful Functions API

[Timothy Bess]

The flink stateful function Python API looks cool, but is there a documented spec for how it communicates with Flink? I'd like to implement an SDK in Haskell if I can.

[Dawid Wysakowicz-2]

Hi,

Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.

To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.

BTW, it would be really cool to see an SDK written in Haskell ;)

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions

[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module

[3] https://github.com/aljoscha/statefun-rust

On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.

[Tzu-Li (Gordon) Tai]

Hi!

Dawid is right, there currently is no developer documentation for the remote request-reply protocol.
One reason for this is that the protocol isn't considered a fully stable user-facing interface yet, and thus not yet properly advertised in the documentation.
However, there are plans to revisit it and announce it as publicly stable in the near future releases.

In the meantime, the Python SDK and Aljoscha's Rust SDK are good reference examples of implementation of the protocol across different languages.
The request body from Flink and expected response body from functions are essentially these [1] Protobuf messages, ToFunction (request) and FromFunction (response).

If you bump into any issues when implementing, please feel free to let us know on the mailing lists as well.
A Haskell SDK is definitely interesting to see implemented :)

Cheers,
Gordon

[1] https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto

On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.

To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.

BTW, it would be really cool to see an SDK written in Haskell ;)

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions

[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module

[3] https://github.com/aljoscha/statefun-rust

On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.

[Timothy Bess]

Hi Gordon and Dawid,

Thanks for the responses! I actually ended up just biting the bullet and digging through the Python implementation to figure it out after I wrote that email. Was more worried about accidentally depending on something internal to the Python API that may not be "spec", but it seemed pretty straightforward when I read through it actually. I made a small implementation over the weekend and got the greeter example working. Not sure how familiar anyone is here with Haskell, but here's the code if you're interested. The example is pretty simple, just have a function that takes messages and sends them out to a counter ID'd by the user's name.

But essentially you interact with the Flink SDK from this one typeclass:

And it essentially just runs through the batches in memory, takes the resulting state, converts it to a "FromFunction" and ships it all back. Haskell has the same sort of green threading concurrency model as Erlang/Go if you're familiar with that, so it was actually very fast when I was testing it.

With these sort of SDKs, do people generally get it merged into some Apache/Flink repo and become official? Or does it just stay on my Github?

Thanks,

Tim

On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

Dawid is right, there currently is no developer documentation for the remote request-reply protocol.
One reason for this is that the protocol isn't considered a fully stable user-facing interface yet, and thus not yet properly advertised in the documentation.
However, there are plans to revisit it and announce it as publicly stable in the near future releases.

In the meantime, the Python SDK and Aljoscha's Rust SDK are good reference examples of implementation of the protocol across different languages.
The request body from Flink and expected response body from functions are essentially these [1] Protobuf messages, ToFunction (request) and FromFunction (response).

If you bump into any issues when implementing, please feel free to let us know on the mailing lists as well.
A Haskell SDK is definitely interesting to see implemented :)

Cheers,
Gordon

[1] https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto

On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.

To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.

BTW, it would be really cool to see an SDK written in Haskell ;)

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions

[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module

[3] https://github.com/aljoscha/statefun-rust

On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.

[Igal Shilman]

Hi Tim,

This is amazing! I hope that it wasn't too much trouble for you to figure out the bits that needed to be implemented for the RequestReply protocol.

Unfortunately I don't know enough Haskell to provide you any meaningful feedback for the SDK itself, not to mention to maintain it at this point.

But what I can suggest is that if you are interested to continue maintaining and evolving this, we can add this to a list of 3rd party SDKs that are developed

by the community. And if eventually enough interest would arise and you would like to contribute this then we can revisit that!

In any case, this is incredible and I'm looking forward to seeing exciting new use cases with Haskell and StateFun :-)

Thanks,

Igal.

On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <[hidden email]> wrote:

Hi Gordon and Dawid,

Thanks for the responses! I actually ended up just biting the bullet and digging through the Python implementation to figure it out after I wrote that email. Was more worried about accidentally depending on something internal to the Python API that may not be "spec", but it seemed pretty straightforward when I read through it actually. I made a small implementation over the weekend and got the greeter example working. Not sure how familiar anyone is here with Haskell, but here's the code if you're interested. The example is pretty simple, just have a function that takes messages and sends them out to a counter ID'd by the user's name.

But essentially you interact with the Flink SDK from this one typeclass:

And it essentially just runs through the batches in memory, takes the resulting state, converts it to a "FromFunction" and ships it all back. Haskell has the same sort of green threading concurrency model as Erlang/Go if you're familiar with that, so it was actually very fast when I was testing it.

With these sort of SDKs, do people generally get it merged into some Apache/Flink repo and become official? Or does it just stay on my Github?

Thanks,

Tim

On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

Dawid is right, there currently is no developer documentation for the remote request-reply protocol.
One reason for this is that the protocol isn't considered a fully stable user-facing interface yet, and thus not yet properly advertised in the documentation.
However, there are plans to revisit it and announce it as publicly stable in the near future releases.

In the meantime, the Python SDK and Aljoscha's Rust SDK are good reference examples of implementation of the protocol across different languages.
The request body from Flink and expected response body from functions are essentially these [1] Protobuf messages, ToFunction (request) and FromFunction (response).

If you bump into any issues when implementing, please feel free to let us know on the mailing lists as well.
A Haskell SDK is definitely interesting to see implemented :)

Cheers,
Gordon

[1] https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto

On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.

To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.

BTW, it would be really cool to see an SDK written in Haskell ;)

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions

[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module

[3] https://github.com/aljoscha/statefun-rust

On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.

[Timothy Bess]

Hi Igal,

Thanks! It wasn't too hard to figure out, the structure makes sense. Will be interesting to see how it evolves once you start lazily fetching state and add other features.

Yeah I expected there may not be many people to review it for mistakes haha, but luckily it's only ~ 245 lines of Haskell so it shouldn't be terribly difficult to maintain. I'm definitely planning to maintain this and will probably start using it at work once we get around to the stateful side of our architecture. I'll add a note to the README that the API is still in flux while I iron out a few things and write a test suite for it. Would definitely be exciting to see Haskell on the list of 3rd party SDKs though! Glad to see Flink moving in this direction, always have been a huge fan of the functionality it provides, but wanted a bit more decoupling from the Flink streaming APIs themselves. This project seems, to me, like someone has essentially figured out how to do polyglot Akka/OTP at large scale with internal ACID state (which is my dream architecture).

Thanks,

Tim

On Mon, Sep 14, 2020 at 1:09 PM Igal Shilman <[hidden email]> wrote:

Hi Tim,

This is amazing! I hope that it wasn't too much trouble for you to figure out the bits that needed to be implemented for the RequestReply protocol.

Unfortunately I don't know enough Haskell to provide you any meaningful feedback for the SDK itself, not to mention to maintain it at this point.

But what I can suggest is that if you are interested to continue maintaining and evolving this, we can add this to a list of 3rd party SDKs that are developed

by the community. And if eventually enough interest would arise and you would like to contribute this then we can revisit that!

In any case, this is incredible and I'm looking forward to seeing exciting new use cases with Haskell and StateFun :-)

Thanks,

Igal.

On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <[hidden email]> wrote:

Hi Gordon and Dawid,

Thanks for the responses! I actually ended up just biting the bullet and digging through the Python implementation to figure it out after I wrote that email. Was more worried about accidentally depending on something internal to the Python API that may not be "spec", but it seemed pretty straightforward when I read through it actually. I made a small implementation over the weekend and got the greeter example working. Not sure how familiar anyone is here with Haskell, but here's the code if you're interested. The example is pretty simple, just have a function that takes messages and sends them out to a counter ID'd by the user's name.

But essentially you interact with the Flink SDK from this one typeclass:

And it essentially just runs through the batches in memory, takes the resulting state, converts it to a "FromFunction" and ships it all back. Haskell has the same sort of green threading concurrency model as Erlang/Go if you're familiar with that, so it was actually very fast when I was testing it.

With these sort of SDKs, do people generally get it merged into some Apache/Flink repo and become official? Or does it just stay on my Github?

Thanks,

Tim

On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

Dawid is right, there currently is no developer documentation for the remote request-reply protocol.
One reason for this is that the protocol isn't considered a fully stable user-facing interface yet, and thus not yet properly advertised in the documentation.
However, there are plans to revisit it and announce it as publicly stable in the near future releases.

In the meantime, the Python SDK and Aljoscha's Rust SDK are good reference examples of implementation of the protocol across different languages.
The request body from Flink and expected response body from functions are essentially these [1] Protobuf messages, ToFunction (request) and FromFunction (response).

If you bump into any issues when implementing, please feel free to let us know on the mailing lists as well.
A Haskell SDK is definitely interesting to see implemented :)

Cheers,
Gordon

[1] https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto

On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.

To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.

BTW, it would be really cool to see an SDK written in Haskell ;)

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions

[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module

[3] https://github.com/aljoscha/statefun-rust

On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.

[Tzu-Li (Gordon) Tai]

Hi,

Just wanted to add that we recently added a page in the docs to list 3rd party SDKs: https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/external.html.
Once you're ready with your Haskell and would want it to be listed on that page, feel free to open a pull request for that :)
You can find the source for the documentation content here: https://github.com/apache/flink-statefun/blob/master/docs/sdk/external.md

Cheers,
Gordon

On Tue, Sep 15, 2020 at 2:21 AM Timothy Bess <[hidden email]> wrote:

Hi Igal,

Thanks! It wasn't too hard to figure out, the structure makes sense. Will be interesting to see how it evolves once you start lazily fetching state and add other features.

Yeah I expected there may not be many people to review it for mistakes haha, but luckily it's only ~ 245 lines of Haskell so it shouldn't be terribly difficult to maintain. I'm definitely planning to maintain this and will probably start using it at work once we get around to the stateful side of our architecture. I'll add a note to the README that the API is still in flux while I iron out a few things and write a test suite for it. Would definitely be exciting to see Haskell on the list of 3rd party SDKs though! Glad to see Flink moving in this direction, always have been a huge fan of the functionality it provides, but wanted a bit more decoupling from the Flink streaming APIs themselves. This project seems, to me, like someone has essentially figured out how to do polyglot Akka/OTP at large scale with internal ACID state (which is my dream architecture).

Thanks,

Tim

On Mon, Sep 14, 2020 at 1:09 PM Igal Shilman <[hidden email]> wrote:

Hi Tim,

This is amazing! I hope that it wasn't too much trouble for you to figure out the bits that needed to be implemented for the RequestReply protocol.

Unfortunately I don't know enough Haskell to provide you any meaningful feedback for the SDK itself, not to mention to maintain it at this point.

But what I can suggest is that if you are interested to continue maintaining and evolving this, we can add this to a list of 3rd party SDKs that are developed

by the community. And if eventually enough interest would arise and you would like to contribute this then we can revisit that!

In any case, this is incredible and I'm looking forward to seeing exciting new use cases with Haskell and StateFun :-)

Thanks,

Igal.

On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <[hidden email]> wrote:

Hi Gordon and Dawid,

Thanks for the responses! I actually ended up just biting the bullet and digging through the Python implementation to figure it out after I wrote that email. Was more worried about accidentally depending on something internal to the Python API that may not be "spec", but it seemed pretty straightforward when I read through it actually. I made a small implementation over the weekend and got the greeter example working. Not sure how familiar anyone is here with Haskell, but here's the code if you're interested. The example is pretty simple, just have a function that takes messages and sends them out to a counter ID'd by the user's name.

But essentially you interact with the Flink SDK from this one typeclass:

And it essentially just runs through the batches in memory, takes the resulting state, converts it to a "FromFunction" and ships it all back. Haskell has the same sort of green threading concurrency model as Erlang/Go if you're familiar with that, so it was actually very fast when I was testing it.

With these sort of SDKs, do people generally get it merged into some Apache/Flink repo and become official? Or does it just stay on my Github?

Thanks,

Tim

On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

Dawid is right, there currently is no developer documentation for the remote request-reply protocol.
One reason for this is that the protocol isn't considered a fully stable user-facing interface yet, and thus not yet properly advertised in the documentation.
However, there are plans to revisit it and announce it as publicly stable in the near future releases.

In the meantime, the Python SDK and Aljoscha's Rust SDK are good reference examples of implementation of the protocol across different languages.
The request body from Flink and expected response body from functions are essentially these [1] Protobuf messages, ToFunction (request) and FromFunction (response).

If you bump into any issues when implementing, please feel free to let us know on the mailing lists as well.
A Haskell SDK is definitely interesting to see implemented :)

Cheers,
Gordon

[1] https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto

On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.

To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.

BTW, it would be really cool to see an SDK written in Haskell ;)

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions

[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module

[3] https://github.com/aljoscha/statefun-rust

On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.

[Timothy Bess]

Hi Gordon,

Awesome thanks for pointing me to that. I'll polish it up a bit and write some tests and submit a PR for that page then. Great job on all this, really love what you guys are doing. Keep up the good work!

Thanks,

Tim

On Tue, Sep 15, 2020 at 4:33 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi,

Just wanted to add that we recently added a page in the docs to list 3rd party SDKs: https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/external.html.
Once you're ready with your Haskell and would want it to be listed on that page, feel free to open a pull request for that :)
You can find the source for the documentation content here: https://github.com/apache/flink-statefun/blob/master/docs/sdk/external.md

Cheers,
Gordon

On Tue, Sep 15, 2020 at 2:21 AM Timothy Bess <[hidden email]> wrote:

Hi Igal,

Thanks! It wasn't too hard to figure out, the structure makes sense. Will be interesting to see how it evolves once you start lazily fetching state and add other features.

Yeah I expected there may not be many people to review it for mistakes haha, but luckily it's only ~ 245 lines of Haskell so it shouldn't be terribly difficult to maintain. I'm definitely planning to maintain this and will probably start using it at work once we get around to the stateful side of our architecture. I'll add a note to the README that the API is still in flux while I iron out a few things and write a test suite for it. Would definitely be exciting to see Haskell on the list of 3rd party SDKs though! Glad to see Flink moving in this direction, always have been a huge fan of the functionality it provides, but wanted a bit more decoupling from the Flink streaming APIs themselves. This project seems, to me, like someone has essentially figured out how to do polyglot Akka/OTP at large scale with internal ACID state (which is my dream architecture).

Thanks,

Tim

On Mon, Sep 14, 2020 at 1:09 PM Igal Shilman <[hidden email]> wrote:

Hi Tim,

This is amazing! I hope that it wasn't too much trouble for you to figure out the bits that needed to be implemented for the RequestReply protocol.

Unfortunately I don't know enough Haskell to provide you any meaningful feedback for the SDK itself, not to mention to maintain it at this point.

But what I can suggest is that if you are interested to continue maintaining and evolving this, we can add this to a list of 3rd party SDKs that are developed

by the community. And if eventually enough interest would arise and you would like to contribute this then we can revisit that!

In any case, this is incredible and I'm looking forward to seeing exciting new use cases with Haskell and StateFun :-)

Thanks,

Igal.

On Mon, Sep 14, 2020 at 3:01 PM Timothy Bess <[hidden email]> wrote:

Hi Gordon and Dawid,

Thanks for the responses! I actually ended up just biting the bullet and digging through the Python implementation to figure it out after I wrote that email. Was more worried about accidentally depending on something internal to the Python API that may not be "spec", but it seemed pretty straightforward when I read through it actually. I made a small implementation over the weekend and got the greeter example working. Not sure how familiar anyone is here with Haskell, but here's the code if you're interested. The example is pretty simple, just have a function that takes messages and sends them out to a counter ID'd by the user's name.

But essentially you interact with the Flink SDK from this one typeclass:

And it essentially just runs through the batches in memory, takes the resulting state, converts it to a "FromFunction" and ships it all back. Haskell has the same sort of green threading concurrency model as Erlang/Go if you're familiar with that, so it was actually very fast when I was testing it.

With these sort of SDKs, do people generally get it merged into some Apache/Flink repo and become official? Or does it just stay on my Github?

Thanks,

Tim

On Mon, Sep 14, 2020 at 4:55 AM Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

Dawid is right, there currently is no developer documentation for the remote request-reply protocol.
One reason for this is that the protocol isn't considered a fully stable user-facing interface yet, and thus not yet properly advertised in the documentation.
However, there are plans to revisit it and announce it as publicly stable in the near future releases.

In the meantime, the Python SDK and Aljoscha's Rust SDK are good reference examples of implementation of the protocol across different languages.
The request body from Flink and expected response body from functions are essentially these [1] Protobuf messages, ToFunction (request) and FromFunction (response).

If you bump into any issues when implementing, please feel free to let us know on the mailing lists as well.
A Haskell SDK is definitely interesting to see implemented :)

Cheers,
Gordon

[1] https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-core/src/main/protobuf/http-function.proto

On Mon, Sep 14, 2020 at 3:46 PM Dawid Wysakowicz <[hidden email]> wrote:

Hi,

Not sure if there is a "developer" documentation for the protocol. I am
cc'ing Igal and Gordon who know better than I if there is one.

To give you some hints though. If I am correct the Python API is
implemented as a so called remote functions [1][2], which communicate
with Flink via HTTP/gRPC. Besides the bundled Python API you can also
use a Rust SDK[3] implemented by my colleague Aljoscha as a reference.

BTW, it would be really cool to see an SDK written in Haskell ;)

Best,

Dawid

[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions

[2]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/modules.html#remote-module

[3] https://github.com/aljoscha/statefun-rust

On 12/09/2020 07:26, Timothy Bess wrote:
> The flink stateful function Python API looks cool, but is there a
> documented spec for how it communicates with Flink? I'd like to
> implement an SDK in Haskell if I can.