Re: Duplicated data when using Externalized Checkpoints in a Flink Highly Available cluster

Posted by F.Amara on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Duplicated-data-when-using-Externalized-Checkpoints-in-a-Flink-Highly-Available-cluster-tp13301p13483.html

Hi Robert,

I have few more questions to clarify.

1) Why do you say printing the values to the standard out would display duplicates even if exactly once works? What is the reason for this? Could you brief me on this?

2) I observed duplicates (by writing to a file) starting from the FlinkKafkaConsumer onwards. Why does this component introduce duplicates? Is it because Kafka guarantees only At-least once delivery at the moment?

Thanks,
Amara