streaming restored state after restart

[Adam Atrea]

Hi,

I'm new to Flink - but after reading the documentation - 

What would be the best approach to stream data from a restored state following a job restart ?

Say I have a MapState that gets populated during streaming with various computed results within a stateful operator.

Upon job restart or on task failure recovery I am looking to stream the data from that restored state downstream so that the operator will replay it.

The data just to be directed to the downstream - at least once - so multiple tasks may submit the same data.

Thanks,

Adam

[Till Rohrmann]

Hi Adam,

you could have a local field in your operator which you initialize with true on creation. Then in the map or process function you could check this field and if true output all buffered state. At last you only need to set this field to false so that any subsequent call to the map/process function won't trigger the emission of the buffered elements in your state.

Cheers,

Till

On Thu, Jun 11, 2020 at 2:58 PM Adam Atrea <[hidden email]> wrote:

Hi,

I'm new to Flink - but after reading the documentation - 

What would be the best approach to stream data from a restored state following a job restart ?

Say I have a MapState that gets populated during streaming with various computed results within a stateful operator.

Upon job restart or on task failure recovery I am looking to stream the data from that restored state downstream so that the operator will replay it.

The data just to be directed to the downstream - at least once - so multiple tasks may submit the same data.

Thanks,

Adam