[Question] enable end2end Kafka exactly once processing

Posted by Eleanore Jin on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Question-enable-end2end-Kafka-exactly-once-processing-tp33253.html

Hi experts, 

My application is using Apache Beam and with Flink to be the runner. My source and sink are kafka topics, and I am using KafkaIO connector provided by Apache Beam to consume and publish. 

I am reading through Beam's java doc: https://beam.apache.org/releases/javadoc/2.16.0/org/apache/beam/sdk/io/kafka/KafkaIO.WriteRecords.html#withEOS-int-java.lang.String-

It looks like Beam does not support Flink Runner for EOS, can someone please shad some lights on how to enable exactly once processing with Apache Beam?

Thanks a lot!

Eleanore