Processing single events for minimum latency

Posted by Pankaj Chand on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Processing-single-events-for-minimum-latency-tp38705.html

Hi all,

What is the recommended way to make a Flink job that processes each event individually as soon as it comes and without waiting for a window, in order to minimize latency in the entire DAG of operators?

For example, here is some sample WordCount code (without windws), followed by some known ways:

 DataStream<WordWithCount> wordCounts = text
            .flatMap(new FlatMapFunction<String, WordWithCount>())
            .keyBy("word")
            .reduce(new ReduceFunction<WordWithCount>());

1. Don't include any TimeWindow/CountWindow function (does this actually achieve what I want?)
2. Use a CountWindow with a count of 1
3. Make a Trigger that fires to process each event when it comes in

I think the above methods only work at the processing level and latency with respect to a single operator, but does not affect the latency of an event in the entire Flink job's DAG since those ways do not affect the buffertimeout value.

Thanks,

Pankaj