Question about exactly-once

[Zhangrucong]

Dear Sir：

I am a beginner of Flink and very interested in “Exactly-once” Recovery Mechanism. I have a question about processing sequence problem of tuples. For example, in Fig 1, process unit A runs JOIN, and the size of sliding window is 4. At the beginning, the state of sliding windows is shown in Fig 2. Before A failed, Tuples came in the order of 1,2,3,4,and the join results are (1,1),(2,2)(2,2)(3,3)(4,4),but after A failed and the state is reset to Snap(x), tuples came in the order of 3,4,1,2. This time the join results are (3,3)(3,3)(4,4)(2,2)(2,2).

Fig 1

Fig 2

I wonder how Flink’s mechanism guarantees the consistency of results or consistency of tuples’ sequence?

 

 

Thank you very much.

[Stephan Ewen]

Hi!

The order of tuples in stream may vary, depending on certain operations. When windows are computed on "processing time" (sometimes called "stream time"), then the result of the windowing depends on the speed of the tuple streams. There are multiple possible outcomes of the computation.

Upon recovery, the system computes one of the possible results, which may in fact be a different result than the original result, if the order of tuples changes.

We are working on integrating "event time" into the streaming system, which defines the windows based on element timestamps. That way, the windows become deterministic - and the recovery computes the exact same as the original computation.

Note that without event time, all windows in all streaming systems are to some extend non-deterministic, as they depend on the speed/order of events in the sources.

For a good introduction to event time, I'll encourage you to read the "MillWheels" paper from Google - a very good read!

Greetings,
Stephan

On Tue, Sep 8, 2015 at 7:05 AM, Zhangrucong <[hidden email]> wrote:

Dear Sir：

I am a beginner of Flink and very interested in “Exactly-once” Recovery Mechanism. I have a question about processing sequence problem of tuples. For example, in Fig 1, process unit A runs JOIN, and the size of sliding window is 4. At the beginning, the state of sliding windows is shown in Fig 2. Before A failed, Tuples came in the order of 1,2,3,4,and the join results are (1,1),(2,2)(2,2)(3,3)(4,4),but after A failed and the state is reset to Snap(x), tuples came in the order of 3,4,1,2. This time the join results are (3,3)(3,3)(4,4)(2,2)(2,2).

Fig 1

Fig 2

I wonder how Flink’s mechanism guarantees the consistency of results or consistency of tuples’ sequence?

 

 

Thank you very much.