Hi,
In old Flink versions (prior to 1.9) that would be the case. If operator D emitted a record to Operator B, but Operator B hasn't yet processed when checkpoint is happening, this record would be lost during recovery. Operator D would be recovered with it's state as it was after emitting this record, but the record would never be delivered to Operator B.
However, since Flink 1.9, iterators are not working with checkpointing on even a deeper level [1] and currently there are no plans to address this issue. We are working on providing better API for iterations in the future.
Best,
Piotrek
pt., 18 cze 2021 o 08:21 Varun Chakravarthy Senthilnathan <
[hidden email]> napisał(a):
Hi All,
We have a sample flow like below :
Operator A
Operator B
Operator C
Operator D
Operator E
We have implemented iterations where the result of code done in Operator C is checked in Operator D and conditionally pushed back into Operator B. Now according to this stackoverflow answer (https://stackoverflow.com/questions/54681200/how-does-flink-treat-checkpoints-and-state-within-iterativestream/54707931#54707931),
events in the loop could be lost in case of failure.
I would like to understand what it means to lose events. Let’s say an event reaches D and it was pushed to operator B and at this instant, the application went down. When the restart happens, will the event be lost as in the progress that
it had made till Operator D be last it was seen and the event cannot be reprocessed again or will it be retriggered from Operator A?
Regards,
Varun.
Locked
