Flink upgrade to Flink-1.12

[耿延杰]

Hi all,

As flink doc says:

https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/upgrading.html#preconditions

We do not support migration for state in RocksDB that was checkpointed using semi-asynchronous mode. In case your old job was using this mode, you can still change your job to use fully-asynchronous mode before taking the savepoint that is used as the basis for the migration.

So, my first question:

Is "semi-asynchronous" means "incremental checkpoint"?

And second question:

If so, assume I'm using flink-1.11 and RocksDB with incremental asynchronous checkpoint as state backend. 

I should: 

1. take a savepoint for old version(flink-1.11), 

2. and change job to use "full asynchronous checkpoint" ,

3. restart old version(flink-1.11) job with new config (full asynchronous checkpoint),

4. then, take a savepoint

5. and finally, stop old version(flink-1.11) and upgrade to flink-1.12

Whether I understand correctly?

Best regards

[Ufuk Celebi]

Thanks for reaching out. Semi-asynchronous does not refer to incremental checkpoints and Savepoints are always triggered as full snapshots (not incremental).

Earlier versions of the RocksDb state backend supported two snapshotting modes, fully and semi-asynchronous snapshots. Semi-asynchronous state snapshots for RocksDb have been removed a long time ago by Aljoscha in https://github.com/apache/flink/pull/2345 (FLINK-4340). The notes you are referencing were added around that time and I'm afraid they might have become mostly obsolete.

I'm pulling in Aljoscha who should be able to give a definitive answer here.

To make a long story short, it should simply work for you to upgrade from 1.11 to 1.12 via a Savepoint.

Cheers,

Ufuk

On Wed, Jan 20, 2021, at 3:58 AM, 耿延杰 wrote:

Hi all,

As flink doc says:

https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/upgrading.html#preconditions

We do not support migration for state in RocksDB that was checkpointed using semi-asynchronous mode. In case your old job was using this mode, you can still change your job to use fully-asynchronous mode before taking the savepoint that is used as the basis for the migration.

So, my first question:

Is "semi-asynchronous" means "incremental checkpoint"?

And second question:

If so, assume I'm using flink-1.11 and RocksDB with incremental asynchronous checkpoint as state backend. 

I should: 

1. take a savepoint for old version(flink-1.11), 

2. and change job to use "full asynchronous checkpoint" ,

3. restart old version(flink-1.11) job with new config (full asynchronous checkpoint),

4. then, take a savepoint

5. and finally, stop old version(flink-1.11) and upgrade to flink-1.12

Whether I understand correctly?

Best regards

[Aljoscha Krettek]

I'm afraid I also don't know more than that. But I agree with Ufuk that
it should just work.

I think the best way would be to try it in a test environment and then
go forward with upgrading the production jobs/cluster.

Best,
Aljoscha

On 2021/01/25 18:59, Ufuk Celebi wrote:

>Thanks for reaching out. Semi-asynchronous does *not* refer to incremental checkpoints and Savepoints are always triggered as full snapshots (not incremental).
>
>Earlier versions of the RocksDb state backend supported two snapshotting modes, fully and semi-asynchronous snapshots. Semi-asynchronous state snapshots for RocksDb have been removed a long time ago by Aljoscha in https://github.com/apache/flink/pull/2345 (FLINK-4340). The notes you are referencing were added around that time and I'm afraid they might have become mostly obsolete.
>
>I'm pulling in Aljoscha who should be able to give a definitive answer here.
>
>To make a long story short, it should simply work for you to upgrade from 1.11 to 1.12 via a Savepoint.
>
>Cheers,
>
>Ufuk
>
>On Wed, Jan 20, 2021, at 3:58 AM, 耿延杰 wrote:
>> Hi all,
>>
>> As flink doc says:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/upgrading.html#preconditions
>>
>>> We do not support migration for state in RocksDB that was checkpointed using `semi-asynchronous` mode. In case your old job was using this mode, you can still change your job to use `fully-asynchronous` mode before taking the savepoint that is used as the basis for the migration.
>>
>> So, my first question:
>> Is "semi-asynchronous" means "incremental checkpoint"?
>>
>> And second question:
>> If so, assume I'm using flink-1.11 and RocksDB with incremental asynchronous checkpoint as state backend.
>> I should:
>> 1. take a savepoint for old version(flink-1.11),
>> 2. and change job to use "full asynchronous checkpoint" ,
>> 3. restart old version(flink-1.11) job with new config (full asynchronous checkpoint),
>> 4. then, take a savepoint
>> 5. and finally, stop old version(flink-1.11) and upgrade to flink-1.12
>>
>> Whether I understand correctly?
>>
>> Best regards