Help with DeltaIteration

[Lydia Ickler]

Hi,

I have a question regarding the Delta Iteration. 

I basically want to iterate as long as the former and the new calculated set are different. Stop if they are the same.

Right now I get a result set that has entries with duplicate „row“ indices which should not be the case.

I guess I am doing something wrong in the iteration.closeWith(intermediate, diffs); Maybe I am sending only parts of the set but for the Multiplication (ProjectJoinResultMapper()) I need the whole DataSet.

Could somebody please hint me in the right direction?

Thanks in advance!

This is what I have right now:

DataSet<Tuple3<Integer, Integer, Double>> initial = matrixA.groupBy(0).sum();

//normalize by maximum value
initial = initial.cross(initial.sum(2)).map(new normalizeByMax());

DeltaIteration<Tuple3<Integer, Integer, Double>,Tuple3<Integer, Integer, Double>> iteration = initial.iterateDelta(initial, 1, 0,1);

DataSet<Tuple3<Integer, Integer, Double>> intermediate = matrixA.join(iteration.getWorkset()).where(1).equalTo(0)
        .map(new ProjectJoinResultMapper()).groupBy(0, 1).sum(2).groupBy(0).sum(2).cross(matrixA.join(iteration.getWorkset()).where(1).equalTo(0)
                .map(new ProjectJoinResultMapper()).groupBy(0, 1).sum(2).groupBy(0).sum(2).max(2)).map(new normalizeByMax());
DataSet<Tuple3<Integer, Integer, Double>> diffs = intermediate.join(iteration.getSolutionSet()).where(0,1).equalTo(0,1).with(new ComponentIdFilter());
DataSet<Tuple3<Integer, Integer, Double>> result = iteration.closeWith(intermediate, diffs);

public static final class ComponentIdFilter implements FlatJoinFunction<Tuple3<Integer, Integer, Double>,Tuple3<Integer, Integer, Double>,Tuple3<Integer, Integer, Double>> {

    public void join(Tuple3<Integer, Integer, Double> candidate, Tuple3<Integer, Integer, Double> old, Collector<Tuple3<Integer, Integer, Double>> out) {

        if(!candidate.f2.equals(old.f2)){
            out.collect(candidate);
        }
    }
}

[Balaji Rajagopalan]

The easier way to debug this would be have prints in the projectjoinresultmapper and see what data you are getting. It is possible your original dataset has duplicate rows ? 

On Thu, Mar 17, 2016 at 6:36 PM, Lydia Ickler <[hidden email]> wrote:

Hi,

I have a question regarding the Delta Iteration. 

I basically want to iterate as long as the former and the new calculated set are different. Stop if they are the same.

Right now I get a result set that has entries with duplicate „row“ indices which should not be the case.

I guess I am doing something wrong in the iteration.closeWith(intermediate, diffs); Maybe I am sending only parts of the set but for the Multiplication (ProjectJoinResultMapper()) I need the whole DataSet.

Could somebody please hint me in the right direction?

Thanks in advance!

This is what I have right now:

DataSet<Tuple3<Integer, Integer, Double>> initial = matrixA.groupBy(0).sum();

//normalize by maximum value
initial = initial.cross(initial.sum(2)).map(new normalizeByMax());

DeltaIteration<Tuple3<Integer, Integer, Double>,Tuple3<Integer, Integer, Double>> iteration = initial.iterateDelta(initial, 1, 0,1);

DataSet<Tuple3<Integer, Integer, Double>> intermediate = matrixA.join(iteration.getWorkset()).where(1).equalTo(0)
        .map(new ProjectJoinResultMapper()).groupBy(0, 1).sum(2).groupBy(0).sum(2).cross(matrixA.join(iteration.getWorkset()).where(1).equalTo(0)
                .map(new ProjectJoinResultMapper()).groupBy(0, 1).sum(2).groupBy(0).sum(2).max(2)).map(new normalizeByMax());
DataSet<Tuple3<Integer, Integer, Double>> diffs = intermediate.join(iteration.getSolutionSet()).where(0,1).equalTo(0,1).with(new ComponentIdFilter());
DataSet<Tuple3<Integer, Integer, Double>> result = iteration.closeWith(intermediate, diffs);

public static final class ComponentIdFilter implements FlatJoinFunction<Tuple3<Integer, Integer, Double>,Tuple3<Integer, Integer, Double>,Tuple3<Integer, Integer, Double>> {

    public void join(Tuple3<Integer, Integer, Double> candidate, Tuple3<Integer, Integer, Double> old, Collector<Tuple3<Integer, Integer, Double>> out) {

        if(!candidate.f2.equals(old.f2)){
            out.collect(candidate);
        }
    }
}