Checkpoints?

[Steve Jerman]

Hi,

Configuration:
Flink 1.2.0
I'm using the Rocks DB backend for checkpointing.

The problem I have is that no checkpoints are being deleted, and my disk is filling up.

Is there configuration for this?

Thanks
Steve

[Till Rohrmann]

Hi Steve,

in the past we had some problems with cleaning up old checkpoints. But this was in 1.1.x. These problems should be fixed by now.

Could you try upgrading to Flink 1.2.1 in order to see whether the problem persists? If this is the case, then it would be great if you could share the JobManager logs on debug log level with us.

How long is your checkpoint interval? Deleting files from HDFS/S3 can take some time and if the checkpoint interval is shorter than this time, then the system won't be able to delete old checkpoints quick enough.

Cheers,

Till

On Fri, Jun 2, 2017 at 5:57 PM, Steve Jerman <[hidden email]> wrote:

Hi,

Configuration:
Flink 1.2.0
I'm using the Rocks DB backend for checkpointing.

The problem I have is that no checkpoints are being deleted, and my disk is filling up.

Is there configuration for this?

Thanks
Steve

[Steve Jerman]

Thanks. Upgraded to 1.2.1 - problem goes away 

Steve

On Jun 2, 2017, at 10:08 AM, Till Rohrmann <[hidden email]> wrote:

Hi Steve,

in the past we had some problems with cleaning up old checkpoints. But this was in 1.1.x. These problems should be fixed by now.

Could you try upgrading to Flink 1.2.1 in order to see whether the problem persists? If this is the case, then it would be great if you could share the JobManager logs on debug log level with us.

How long is your checkpoint interval? Deleting files from HDFS/S3 can take some time and if the checkpoint interval is shorter than this time, then the system won't be able to delete old checkpoints quick enough.

Cheers,

Till

On Fri, Jun 2, 2017 at 5:57 PM, Steve Jerman <[hidden email]> wrote:

Hi,

Configuration:
Flink 1.2.0
I'm using the Rocks DB backend for checkpointing.

The problem I have is that no checkpoints are being deleted, and my disk is filling up.

Is there configuration for this?

Thanks
Steve