Re: Cannot see all events in window apply() for big input

Posted by Till Rohrmann on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Cannot-see-all-events-in-window-apply-for-big-input-tp9945p9952.html

And this other job also performs a window operation based on event time?

What do you mean with “I have a doubt is the necessary parallelism for window operation if reprocessing a skew input from Kafka”?

Also be aware that the windowAll operation is executed with a dop of 1, making it effectively a non-parallel operation.

Have you tried switching to the latest Flink version for the tests?

Cheers,
Till

​

On Mon, Nov 7, 2016 at 5:43 PM, Sendoh <[hidden email]> wrote:

Hi Till.

Thank you for suggesting. We know the timestamp is correct because another
Flink job is running with the three topics correctly. We also know the
operators work well before window apply() because we check the result before
window apply().

What currently I have a doubt is the necessary parallelisms for window
operation if reprocessing a skew input from Kafka because it works with
fewer events, and small topics always appear while big topics disappear.

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Cannot-see-all-events-in-window-apply-for-big-input-tp9945p9950.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.