Hi again,
I had a bug in my logic. It works as expected (which is perfect).
So maybe for others:
Problem:
- execute superstep-dependent UDFs on datasets which do not have access
to the iteration context
Solution:
- add dummy element to the working set (W) at the beginning of the step
function
- extract dummy from W using a filter function
- convert dummy into DataSet<Integer> (superstep) using a map function
- broadcast that 1-element dataset to the UDFs applied on the "external"
datasets
- filter non-dummy elements (if necessary) and continue step function
Note, that it should also work with cross instead of broadcasting, I did
not test which way works faster, yet.
Apologies if anyone thought about this when it was my error in the end :)
Cheers,
Martin
On 29.05.2016 14:05, Martin Junghanns wrote:
> Hi everyone,
>
> In a step-function (bulk) I'd like to join the working set W
> with another data set T. The join field of T depends on
> the current super step. Unfortunately, W has no access
> to the iteration runtime context.
>
> I tried to extract the current superstep at the beginning of
> the step function and broadcasted it to a UDF applied on T
> (which sets the correct value join field) and perform the join
> always on the same fields. Unfortunately, this does not seem
> to work either.
>
> I could work around that by replicating the elements of T and
> join multiple times but this does not scale very well.
>
> Any suggestion would be appreciated.
>
> Cheers and thank you,
>
> Martin
>
Locked
