Flink stream job change and recovery

[Renjie Liu]

Hi, all:

It seems that flink's checkpoint mechanism saves state per partition. However, if I want to change configuration of the job and resubmit the job, I can not recover from last checkpoint, right? Say I change the parallelism of kafka consumer and partitions assigned to each subtask will be different, then I'm not able to restart from last checkpoint, right?

--

[Fabian Hueske-2]

Yes, that is true for Flink 1.1.x.

The upcoming Flink 1.2.0 release will be able to restore jobs from savepoints with different parallelism.

2016-11-04 11:24 GMT+01:00 Renjie Liu <[hidden email]>:

Hi, all:

It seems that flink's checkpoint mechanism saves state per partition. However, if I want to change configuration of the job and resubmit the job, I can not recover from last checkpoint, right? Say I change the parallelism of kafka consumer and partitions assigned to each subtask will be different, then I'm not able to restart from last checkpoint, right?

--

[Renjie Liu]

Hi, Fabian:

Will the checkpointed state be restored between streaming jobs?

On Fri, Nov 4, 2016 at 6:47 PM Fabian Hueske <[hidden email]> wrote:

Yes, that is true for Flink 1.1.x.

The upcoming Flink 1.2.0 release will be able to restore jobs from savepoints with different parallelism.

2016-11-04 11:24 GMT+01:00 Renjie Liu <[hidden email]>:

Hi, all:

It seems that flink's checkpoint mechanism saves state per partition. However, if I want to change configuration of the job and resubmit the job, I can not recover from last checkpoint, right? Say I change the parallelism of kafka consumer and partitions assigned to each subtask will be different, then I'm not able to restart from last checkpoint, right?

--

--

[Fabian Hueske-2]

Hi,

savepoints are essentially checkpoints with some additional metadata.

So changing the parallelism in 1.2 will work as follows:

- take a savepoint (i.e., an explicitly triggered checkpoint) and stop the job

- restart the application from the savepoint with new parallelism

The application will not lose any state.

Best, Fabian

2016-11-04 12:26 GMT+01:00 Renjie Liu <[hidden email]>:

Hi, Fabian:

Will the checkpointed state be restored between streaming jobs?

On Fri, Nov 4, 2016 at 6:47 PM Fabian Hueske <[hidden email]> wrote:

Yes, that is true for Flink 1.1.x.

The upcoming Flink 1.2.0 release will be able to restore jobs from savepoints with different parallelism.

2016-11-04 11:24 GMT+01:00 Renjie Liu <[hidden email]>:

Hi, all:

It seems that flink's checkpoint mechanism saves state per partition. However, if I want to change configuration of the job and resubmit the job, I can not recover from last checkpoint, right? Say I change the parallelism of kafka consumer and partitions assigned to each subtask will be different, then I'm not able to restart from last checkpoint, right?

--

--