“feedback loop” and checkpoints in itearative streams

[Rinat]

Hi mates, for some reason, it's necessary to create a feedback look in my streaming application.

The best choice to implement it was iterative stream, but at the moment of job implementation (flink version is 1.6.1) it wasn't checkpointed.

So I decided to write this output into kafka. 

As I see, there is no changes at this moment (from official docs)

> Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.

Does the community have any plans to fix this behaviour ?

thx !

[Igal Shilman]

Hi,

I don't know what is the status of iterations at the moment, and whatever the community has plans to work at that,

But I would like to point you to Flink Stateful Functions [1], a recent contribution to Apache Flink that allows building 

applications composed of stateful functions that can invoke (message) each other in an arbitrary order (not necessarily a DAG)

Kind regards,

Igal.

[1] https://github.com/apache/flink-statefun

On Sat, Apr 4, 2020 at 12:54 PM Sharipov, Rinat <[hidden email]> wrote:

Hi mates, for some reason, it's necessary to create a feedback look in my streaming application.

The best choice to implement it was iterative stream, but at the moment of job implementation (flink version is 1.6.1) it wasn't checkpointed.

So I decided to write this output into kafka. 

As I see, there is no changes at this moment (from official docs)

> Please note that records in flight in the loop edges (and the state changes associated with them) will be lost during failure.

Does the community have any plans to fix this behaviour ?

thx !