Re: Cannot see all events in window apply() for big input

Posted by Hung on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Cannot-see-all-events-in-window-apply-for-big-input-tp9945p9977.html

Yes. the other job performs event time window and we tried 1.2-SNAPSHOT and 1.1.3. The old version 1.0.3 we lost much much less data. We tried both windowAll() and keyBy() window() already, and tried very tiny lag and window(1 millisecond).

My doubt comes from smaller input works while bigger input has issue (events disappear).

For example, eventA disappears with timestamp after Oct.24 and appears again after around 5 minutes with timestamp at Nov.08, and all events in between(10-25 to 11-07) are missing. The output of the window gets stuck for around 5 minutes. However, if this flink job only reads eventA, we can see all of them.

It looks like data is stuck in that operator and the watermark of that event which should trigger the window comes too late, when there is a lot of data, or?

Best,

Sendoh