Externalized Checkpoints vs Periodic Checkpoints

[Yassine MARZOUGUI]

Hi all,

Could someone clarify the difference between externalized checkpoints[1] and regular periodic checkpoints[2]?

Moreover, I have a question regarding the retention of checkpoints: For regular checkpoints, does the last checkpoint discard the previous ones? If yes, is that the case too for the externalized checkpoints? Thank you.

Best,

Yassine

[1] : https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/checkpoints.html

[2] : https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/checkpointing.html

[Till Rohrmann]

Hi Yassine,

a periodic checkpoint is checkpoint which will be triggered periodically by Flink. The checkpoint itself can have multiple properties and one of them is whether the checkpoint is externalized or not. 

An externalized checkpoint is a checkpoint for which Flink writes the meta information into a target directory. In contrast to that, for a non-externalized checkpoint Flink will store the checkpoint meta information only in memory. The former has the advantage that you don't lose the checkpoints if you shutdown your cluster. They behave similar to savepoints and in fact savepoints are externalized checkpoints with some more properties.

At the moment, Flink's checkpoint coordinator only retains the last successfully completed checkpoint. This means that whenever a new checkpoint completes then the last completed checkpoint will be discarded. This also applies to externalized checkpoints.

Cheers,

Till

On Wed, Feb 1, 2017 at 2:03 PM, Yassine MARZOUGUI <[hidden email]> wrote:

Hi all,

Could someone clarify the difference between externalized checkpoints[1] and regular periodic checkpoints[2]?

Moreover, I have a question regarding the retention of checkpoints: For regular checkpoints, does the last checkpoint discard the previous ones? If yes, is that the case too for the externalized checkpoints? Thank you.

Best,

Yassine

[1] : https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/checkpoints.html

[2] : https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/checkpointing.html

[Yassine MARZOUGUI]

Thank you Till for the clarification, that was helpful.

Best,

Yassine

2017-02-02 15:31 GMT+01:00 Till Rohrmann <[hidden email]>:

Hi Yassine,

a periodic checkpoint is checkpoint which will be triggered periodically by Flink. The checkpoint itself can have multiple properties and one of them is whether the checkpoint is externalized or not. 

An externalized checkpoint is a checkpoint for which Flink writes the meta information into a target directory. In contrast to that, for a non-externalized checkpoint Flink will store the checkpoint meta information only in memory. The former has the advantage that you don't lose the checkpoints if you shutdown your cluster. They behave similar to savepoints and in fact savepoints are externalized checkpoints with some more properties.

At the moment, Flink's checkpoint coordinator only retains the last successfully completed checkpoint. This means that whenever a new checkpoint completes then the last completed checkpoint will be discarded. This also applies to externalized checkpoints.

Cheers,

Till

On Wed, Feb 1, 2017 at 2:03 PM, Yassine MARZOUGUI <[hidden email]> wrote:

Hi all,

Could someone clarify the difference between externalized checkpoints[1] and regular periodic checkpoints[2]?

Moreover, I have a question regarding the retention of checkpoints: For regular checkpoints, does the last checkpoint discard the previous ones? If yes, is that the case too for the externalized checkpoints? Thank you.

Best,

Yassine

[1] : https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/checkpoints.html

[2] : https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/checkpointing.html