Re: Context-specific step function in Iteration

Posted by Martin Junghanns-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Context-specific-step-function-in-Iteration-tp7234p7244.html

Hi again,

I had a bug in my logic. It works as expected (which is perfect).

So maybe for others:

Problem:
- execute superstep-dependent UDFs on datasets which do not have access
to the iteration context

Solution:
- add dummy element to the working set (W) at the beginning of the step
function
- extract dummy from W using a filter function
- convert dummy into DataSet<Integer> (superstep) using a map function
- broadcast that 1-element dataset to the UDFs applied on the "external"
datasets
- filter non-dummy elements (if necessary) and continue step function

Note, that it should also work with cross instead of broadcasting, I did
not test which way works faster, yet.

Apologies if anyone thought about this when it was my error in the end :)

Cheers,
Martin


On 29.05.2016 14:05, Martin Junghanns wrote:

> Hi everyone,
>
> In a step-function (bulk) I'd like to join the working set W
> with another data set T. The join field of T depends on
> the current super step. Unfortunately, W has no access
> to the iteration runtime context.
>
> I tried to extract the current superstep at the beginning of
> the step function and broadcasted it to a UDF applied on T
> (which sets the correct value join field) and perform the join
> always on the same fields. Unfortunately, this does not seem
> to work either.
>
> I could work around that by replicating the elements of T and
> join multiple times but this does not scale very well.
>
> Any suggestion would be appreciated.
>
> Cheers and thank you,
>
> Martin
>