Re: Cannot see all events in window apply() for big input

Posted by Till Rohrmann on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Cannot-see-all-events-in-window-apply-for-big-input-tp9945p9993.html

Flink does not support per key watermarks or type sensitive watermarks. The underlying assumption is that you have a global watermark which defines the progress wrt to event time in your topology.

The easiest way would be to have an input which has a monotonically increasing timestamp. Alternatively you can define the maximum lag between the watermark and the timestamp and then generate watermarks with w = timestamp - maxLag. That way you allow elements to be out of order for a certain amount of event time.

Cheers,

Till

On Tue, Nov 8, 2016 at 5:02 PM, Sendoh <[hidden email]> wrote:

Thank you for confirming.

What would you think an efficient way not having global watermark? The
following logic fails to build Watermark per KeyStream:
jsonStreams.keyBy(new JsonKeySelector()).assignTimestampsAndWatermarks(new
JsonWatermark()).keyBy(JsonKeySelector()).window(....

So, using split(), or implementing an event type recognized
AssignerWithPeriodicWatermarks along with custom EventTimeTrigger would be
the solution?

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-see-all-events-in-window-apply-for-big-input-tp9945p9988.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.