Re: Data point goes missing within iteration

Posted by Paris Carbone on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Data-point-goes-missing-within-iteration-tp7776p8043.html

This is possibly related to the way the queue between StreamIterationTail and Head is currently implemented.

I think this part is a bit prone to records loss when things get wacky and backpressure kicks in (but at least it avoids deadlocks, right?).

I don’t have the time availability to look into the code right now but I am going to focus on a progress/loops by September, hopefully with a FLIP, which solves that part as well. 

If  this is urgent, please go ahead and check this now, I think that queue timeouts cause this...

Paris

PS: on my yet incomplete PR (I know I know) I basically disabled queue polling timeouts since the checkpoint overhead on the StreamIterationHead almost always led to record loss.

https://github.com/apache/flink/pull/1668

On 20 Jul 2016, at 11:57, Maximilian Michels <[hidden email]> wrote:

CC Gyula and Paris in case they might want to help out.

On Tue, Jul 19, 2016 at 11:43 AM, Biplob Biswas
<[hidden email]> wrote:

Hi Ufuk,

Thanks for the update, is there any known way to fix this issue? Any
workaround that you know of, which I can try?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Data-point-goes-missing-within-iteration-tp7776p8015.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.