Re: Making job fail on Checkpoint Expired?
Posted by
Robin Cassan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Making-job-fail-on-Checkpoint-Expired-tp34051p34066.html
Hi Congxian,
Thanks for confirming! The reason I want this behavior is because we are currently investigating issues with checkpoints that keep getting timeouts after the job has been running for a few hours. We observed that, after a few timeouts, if the job was being restarted because of a lost TM for example, the next checkpoints would be working for a few more hours. However, if the job continues running and consuming more data, the next checkpoints will be even bigger and the chances of them completing in time are getting even thinner.
Crashing the job is not a viable solution I agree, but it would allow us to generate data during the time we investigate the root cause of the timeouts.
I believe that having the option to make the job restart after a few checkpoint timeouts would still help to avoid the snowball effect of incremental checkpoints being bigger and bigger if the checkpoints keep getting expired.
I'd love to get your opinion on this!
Thanks,
Robin
Currently, only checkpoint declined will be counted into `continuousFailureCounter`.
Could you please share why do you want the job to fail when checkpoint expired?
Hi Robin,
this is a very good observation and maybe even unintended behavior.
Maybe Arvid in CC is more familiar with the checkpointing?
Regards,
Timo
On 02.04.20 15:37, Robin Cassan wrote:
> Hi all,
>
> I am wondering if there is a way to make a flink job fail (not cancel
> it) when one or several checkpoints have failed due to being expired
> (taking longer than the timeout) ?
> I am using Flink 1.9.2 and have set
> `*setTolerableCheckpointFailureNumber(1)*` which doesn't do the trick.
> Looking into the CheckpointFailureManager.java class, it looks like this
> only works when the checkpoint failure reason is
> `*CHECKPOINT_DECLINED*`, but the number of failures isn't incremented on
> `*CHECKPOINT_EXPIRED*`.
> Am I missing something?
>
> Thanks!