Re: Starting a job that does not use checkpointing from a savepoint is broken ?
Posted by
Eron Wright on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Starting-a-job-that-does-not-use-checkpointing-from-a-savepoint-is-broken-tp17874p17875.html
To restate the issue:
When checkpointing is disabled, the Flink Kafka Consumer relies on the periodic offsets that are committed to the broker by the internal Kafka client. Such a job would, upon restart, continue from the committed offsets. However, in the situation that the job is restored from a savepoint, then the offsets within the savepoint supercede the broker-based offsets.
It seems a bit unusual to use the savepoint feature on a job that doesn't have checkpointing enabled. Makes me wonder whether `StreamExecutionEnvironment::enableCheckpointing`, is best understood as enabling +periodic+ checkpointing.
The docs say that the periodic offset commit feature is not intended for fault tolerance, implying to me that you should use Flink's checkpointing feature. A great reason to use Flink checkpointing is to capture the intermediate state of the job, such as window state, in addition to the consumer offsets.
I hope this helps,
Eron