Re: Controlling the amount of checkpoint files
Posted by
Boris Lublinsky on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Controlling-the-amount-of-checkpoint-files-tp28096p28267.html
So if you have externalized checkpoints, they are never purged?
The issue is that if your state size is rather large, this seems to be the only option.
Hey Boris,
I think the problem is that you are using externalized checkpoints:
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
Your checkpoints are retained in both failure and cancellation cases, so the checkpoint files with grow indefinitely
Hi Boris
For the configure you gave, you can try to reduce the parallelism of the operator which contains states.
Here is code enabling checkpointing
// Enable checkpointing
env.enableCheckpointing(60000 ) // 1 min
env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
val checkpointingBackend = new FsStateBackend("file:///flink/checkpoints", true)
env.setStateBackend(checkpointingBackend)
Hi
Which state backed(Heap or RocksDB) and checkpoint mode (fullsnapshot or increment) do you use?
Is there a way to limit the amount of checkpoint file?
The parameter that I set : state.checkpoints.num-retained: 5
does not seem to have any effect. Is there anything else I can set to prevent infinite growth of checkpointing info?