Can a master class control the superstep in Flink Spargel ?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Can a master class control the superstep in Flink Spargel ?

Hung
Hi,

Would it be available to control the supersteps in Flink Spargel?
For example, a master controls the basic graph algorithm having 5 phases and the master can switch between the phases.

In the given example of Spargel those are send msg and update msg sequentially.
Would it be possible to switch to a specific super step?


Reply | Threaded
Open this post in threaded view
|

Re: Can a master class control the superstep in Flink Spargel ?

Vasiliki Kalavri
Hi,

currently, there is no such built-in master compute class, but you can easily have the equivalent functionality it as follows:

- If your algorithm has a fixed pattern of superstep types, e.g. an initialization superstep, a main phase and a finalization superstep, then you can simply chain these types of Spargel iterations together, by feeding the output of one as input to the next.

For example:

Graph inputGraph = ...
Graph graph1 = inputGraph.runVertexCentricIteration(new UpdateType1(), new MessangerType1(), maxIter1);
Graph graph2 = graph1.runVertexCentricIteration(new UpdateType2(), new MessangerType2(), maxIter2);
Graph graph3 = graph2.runVertexCentricIteration(new UpdateType3(), new MessangerType3(), maxIter3);

and so on.

- If your algorithm needs to switch between superstep types based on some control flow that is not defined beforehand, you can emulate the master functionality by using an iteration aggregator. An aggregator can accumulate values from all vertices during an iteration and then, the aggregated value can be read in the beginning of the next iteration. This way you could keep track of different phases inside your algorithm.

For example, your VertexUpdateFunction could look like this:

VertexUpdater extends VertexUpdateFunction {
int phase;
@Override
public void preSuperstep() {
phase = this.getIterationAggregator("phase.aggregator");
}

@Override
public void updateVertex(key, value, messages) {
if (phase == 1) {
// do type 1 superstep
....
}
else if (phase ==2) {
// do type 2 superstep
....
}
else ...
}
}

Cheers,
V.

On 14 February 2015 at 22:59, HungChang <[hidden email]> wrote:
Hi,

Would it be available to control the supersteps in Flink Spargel?
For example, a master controls the basic graph algorithm having 5 phases and
the master can switch between the phases.

In the given example of Spargel those are send msg and update msg
sequentially.
Would it be possible to switch to a specific super step?






--
View this message in context: http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Can-a-master-class-control-the-superstep-in-Flink-Spargel-tp733.html
Sent from the Apache Flink (Incubator) User Mailing List archive. mailing list archive at Nabble.com.