Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

Posted by min.tan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Lantency-caised-Flink-Checkpoint-on-EXACTLY-ONCE-mode-tp27096.html

Hi,

 

I have a simple data pipeline of a Kafka source, a flink map operator and  a Kafka sink.

 

I have a quick question about latency caused by the checkpoint on the exactly once mode.

 

Due to the changes are committed and visible on a checkpoint completion, so the latency could be as long as that length of checkpoint interval e.g. 5seconds?

 

Is my understanding correct?

 

If I use the at least mode, there will be this addition on latency.  More interestingly, the flink document https://ci.apache.org/projects/flink/flink-docs-release-1.7/internals/stream_checkpointing.html indicate that "dataflows with only embarrassingly parallel streaming operations (map(), flatMap(), filter(), …) actually give exactly once guarantees even in at least once mode."

 

Unfortunately, I have been not able to achieve the exactly once with the at least once. Do I need more settings than I have with the exactly once mode?

 

Many thanks for the advises in advance.

 

Min