Context-specific step function in Iteration

Posted by Martin Junghanns-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Context-specific-step-function-in-Iteration-tp7234.html

Hi everyone,

In a step-function (bulk) I'd like to join the working set W
with another data set T. The join field of T depends on
the current super step. Unfortunately, W has no access
to the iteration runtime context.

I tried to extract the current superstep at the beginning of
the step function and broadcasted it to a UDF applied on T
(which sets the correct value join field) and perform the join
always on the same fields. Unfortunately, this does not seem
to work either.

I could work around that by replicating the elements of T and
join multiple times but this does not scale very well.

Any suggestion would be appreciated.

Cheers and thank you,

Martin