Can rocksDBBackend handle rescaling?

[bupt_ljy]

Hi, all

   I’m using Flink 1.6 and I’m goint to use RocksDBBackend as the checkpoint state backend. Before that, I need to make sure that the checkpoint can handle rescale. From the docs on Apache Flink https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html, it says that the checkpoint do not support Flink’s rescaling, however, from the source code, the rescaling logic seems not to be related to the state backend if the substask’s states are restored from files successfully.

   Could anyone explain this? Or tell me if I miss something?

   

Best, Jiayi Liao

 

[杨力]

A checkpoint cannot be used for rescaling. You should take a savepoint, stop the application, and resume from the savepoint with a new parallelism.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html

On Fri, Sep 14, 2018 at 1:50 AM 廖嘉逸 <[hidden email]> wrote:

Hi, all

   I’m using Flink 1.6 and I’m goint to use RocksDBBackend as the checkpoint state backend. Before that, I need to make sure that the checkpoint can handle rescale. From the docs on Apache Flink https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html, it says that the checkpoint do not support Flink’s rescaling, however, from the source code, the rescaling logic seems not to be related to the state backend if the substask’s states are restored from files successfully.

   Could anyone explain this? Or tell me if I miss something?

   

Best, Jiayi Liao

 

[bupt_ljy]

Thanks for your answer, but I still have some questions.

Could you tell me why the checkpoint cannot be used for rescaling? From my perspective, the difference between checkpoint and savepoint is nullable externalSavepointLocation. And from this doc: https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html, it talked about rescaling based on the checkpointing. Do I confuse these two things?

 Original Message 

Sender: 杨力<[hidden email]>

Recipient: 廖嘉逸<[hidden email]>

Cc: user<[hidden email]>

Date: Friday, Sep 14, 2018 02:03

Subject: Re: Can rocksDBBackend handle rescaling?

A checkpoint cannot be used for rescaling. You should take a savepoint, stop the application, and resume from the savepoint with a new parallelism.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html

On Fri, Sep 14, 2018 at 1:50 AM 廖嘉逸 <[hidden email]> wrote:

Hi, all

   I’m using Flink 1.6 and I’m goint to use RocksDBBackend as the checkpoint state backend. Before that, I need to make sure that the checkpoint can handle rescale. From the docs on Apache Flink https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html, it says that the checkpoint do not support Flink’s rescaling, however, from the source code, the rescaling logic seems not to be related to the state backend if the substask’s states are restored from files successfully.

   Could anyone explain this? Or tell me if I miss something?

   

Best, Jiayi Liao

 

[Andrey Zagrebin]

Hi Jiayi Liao,

I have answered the similar question here:

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Why-documentation-always-say-checkpoint-does-not-support-Flink-specific-features-like-rescaling-td23982.html#a23983

This link might be also useful:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Can-I-only-use-checkpoints-instead-of-savepoints-in-production-td22538.html

I would recommend to use explicit savepoints at the moment for rescaling.

Best,

Andrey

On 13 Sep 2018, at 20:29, bupt_ljy <[hidden email]> wrote:

Thanks for your answer, but I still have some questions.

Could you tell me why the checkpoint cannot be used for rescaling? From my perspective, the difference between checkpoint and savepoint is nullable externalSavepointLocation. And from this doc: https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html, it talked about rescaling based on the checkpointing. Do I confuse these two things?

 Original Message 

Sender: 杨力<[hidden email]>

Recipient: 廖嘉逸<[hidden email]>

Cc: user<[hidden email]>

Date: Friday, Sep 14, 2018 02:03

Subject: Re: Can rocksDBBackend handle rescaling?

A checkpoint cannot be used for rescaling. You should take a savepoint, stop the application, and resume from the savepoint with a new parallelism.

https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html

On Fri, Sep 14, 2018 at 1:50 AM 廖嘉逸 <[hidden email]> wrote:

Hi, all

   I’m using Flink 1.6 and I’m goint to use RocksDBBackend as the checkpoint state backend. Before that, I need to make sure that the checkpoint can handle rescale. From the docs on Apache Flink https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html, it says that the checkpoint do not support Flink’s rescaling, however, from the source code, the rescaling logic seems not to be related to the state backend if the substask’s states are restored from files successfully.

   Could anyone explain this? Or tell me if I miss something?

   

Best, Jiayi Liao