Hadoop's cleanup() in Flink mapper

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Hadoop's cleanup() in Flink mapper

Hung
This post was updated on .
Hi,

Would it be possible to use cleanup() in Map() in Flink?
Some similar functions are found.
http://flink.apache.org/docs/0.6-incubating/api/java/org/apache/flink/runtime/operators/MatchDriver.html

My goal is to make the code wait until the task is complete to output the results,
and cleanup() runs after the last map function runs
Reply | Threaded
Open this post in threaded view
|

Re: Hadoop's cleanup() in Flink mapper

Hung
This post was updated on .
Already found close() in Rich functions which looks similar.
However, cleanup(Context context) can have one parameter and postpone output.
Is there a way to use Collector in close()?
Ex.
close(Collector collector){
collector.collect(new Tuple3<Long, Long, Long>((long) 1, key,
                                                value))
}