I’m working with a kafka environment where I’m limited to 100 partitions @ 1GB log.retention.bytes each. I’m looking to implement exactly once processing from this kafka source to a S3 sink.
If I have understood correctly, Flink will only commit the kafka offsets when the data has been saved to S3.
Have I understood correctly that for Flink checkpoints and exactly once to work, the assumption is that the number and size of partitions (log.retention.bytes) in kafka are sufficient that should a checkpoint need to be rolled back, the data still exists in kafka (I.e. it hasn’t been over-written by new data)?
If the above is true, and I am using a DateTimeBucketer the bucket sizes will directly influence how big the partitions should be in kafka, because larger buckets will result in less frequent commits of the offsets?
Many thanks,
Chris
Locked
