Thanks for the quick response.
We are using FsStateBackend, and I did see checkpoint files and directories in the EFS mounted directory.
We do monitor backpressure through rest api periodically and we do not see any.
From: Yun Gao <[hidden email]>
Sent: Monday, December 21, 2020 10:40 AM
To: Colletta, Edward <[hidden email]>; [hidden email]
Subject: Re: checkpointing seems to be throttled.
This email is from an external source -
exercise caution regarding links and attachments.
Hi Edward,
For the second issue, have you also set the statebackend type? I'm asking so because except for the default heap statebackend, other statebackends should
throws exception if the state.checkpoint.dir is not set. Since heap statebackend stores all the snapshots in the JM's memory, it could not be recovered after JM failover, which makes it not suitable for production usage. Therefore, if used in production env
then it might better to switch to statebackend like rocksdb.
For the checkpoint timeout, AFAIK there should be no large changes after 1.9.2. There may be different issues for checkpoint timeout, and one possible one
might be there are back-pressure due to some operator could not process its records in time, which would block the checkpoints. I think you might check the back-pressure [1] first, and if there is indeed back pressure, then you might try unaligned checkpoints
or solve the back pressure by increasing the parallelism of slow operators.
Best,
Yun