Flink Iterations Ordering

[David Kim]

Hello all,

I read the documentation at [1] on iterations and had a question on whether an assumption is safe to make.

As partial solutions are continuously looping through the step function, when new elements are added as iteration inputs will the insertion order of all of the elements be preserved?

Example:

Current partial solutions going through step function are: A, B, C.

At a later time we add (in order) elements D, E.

Will the iteration result always be A,B,C,D,E?

References:

[1] https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html

[Aljoscha Krettek]

Hi,

as I understand it the order of elements will not be preserved across iteration supersets. But maybe some-one else knows more.

Cheers,

Aljoscha

On Thu, 28 Apr 2016 at 00:23 David Kim <[hidden email]> wrote:

Hello all,

I read the documentation at [1] on iterations and had a question on whether an assumption is safe to make.

As partial solutions are continuously looping through the step function, when new elements are added as iteration inputs will the insertion order of all of the elements be preserved?

Example:

Current partial solutions going through step function are: A, B, C.

At a later time we add (in order) elements D, E.

Will the iteration result always be A,B,C,D,E?

References:

[1] https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html

[Stephan Ewen]

Hi!

The order in which the elements arrive in an iteration HEAD is the order in which the last operator in the loop (the TAIL) produces them. If that is a deterministic ordering (because of a sorted reduce, for example), then you should be able to rely on the order.

Otherwise, the order of elements can always change every time elements are re-distributed across tasks (for example during a key-based shuffle, a broadcast, or a change of parallelism).

Greetings,

Stephan

On Mon, May 2, 2016 at 2:50 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

as I understand it the order of elements will not be preserved across iteration supersets. But maybe some-one else knows more.

Cheers,

Aljoscha

On Thu, 28 Apr 2016 at 00:23 David Kim <[hidden email]> wrote:

Hello all,

I read the documentation at [1] on iterations and had a question on whether an assumption is safe to make.

As partial solutions are continuously looping through the step function, when new elements are added as iteration inputs will the insertion order of all of the elements be preserved?

Example:

Current partial solutions going through step function are: A, B, C.

At a later time we add (in order) elements D, E.

Will the iteration result always be A,B,C,D,E?

References:

[1] https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html