Cancel the flink task and restore from checkpoint ,can I change the flink operator's parallelism

[LakeShen]

Hi community,

      I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure.

     Thanks to your reply.

Best wishes,

LakeShen

[Sivaprasanna]

I think you can modify the operator’s parallelism. It is only if you have set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost.

-

Sivaprasanna 

On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]> wrote:

Hi community,

      I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure.

     Thanks to your reply.

Best wishes,

LakeShen

[Piotr Nowojski-3]

Hi,

Yes, you can change the parallelism. One thing that you can not change is “max parallelism”.

Piotrek

On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]> wrote:

I think you can modify the operator’s parallelism. It is only if you have set maxParallelism, and while restoring from a checkpoint, you shouldn’t modify the maxParallelism. Otherwise, I believe the state will be lost.

-

Sivaprasanna 

On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]> wrote:

Hi community,

      I have a question is that I cancel the flink task and retain the checkpoint dir, then restore from the checkpoint dir ,can I change the flink operator's parallelism,in my thoughts, I think I can't change the flink operator's parallelism,but I am not sure.

     Thanks to your reply.

Best wishes,

LakeShen

[Till Rohrmann]

If you want to change the max parallelism then you need to take a savepoint and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html

Cheers,

Till

On Sat, Mar 14, 2020 at 4:07 AM LakeShen <[hidden email]> wrote:

Hi Eleanore , if you resume from savepoint , you can't change the flink
operator's max parallelism .

Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道：

> Hi Piotr,
> Does this also apply to savepoint? (meaning the max parallelism should not
> change for job resume from savepoint?)
>
> Thanks a lot!
> Eleanore
>
> On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Yes, you can change the parallelism. One thing that you can not change is
> > “max parallelism”.
> >
> > Piotrek
> >
> > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]>
> > wrote:
> > >
> > > I think you can modify the operator’s parallelism. It is only if you
> > have set maxParallelism, and while restoring from a checkpoint, you
> > shouldn’t modify the maxParallelism. Otherwise, I believe the state will
> be
> > lost.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]
> > <mailto:[hidden email]>> wrote:
> > > Hi community,
> > >       I have a question is that I cancel the flink task and retain the
> > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > flink operator's parallelism,in my thoughts, I think I can't change the
> > flink operator's parallelism,but I am not sure.
> > >      Thanks to your reply.
> > >
> > > Best wishes,
> > > LakeShen
> >
> >
>

[Till Rohrmann]

Let us know if you should run into any problems. The state processor API is still young and could benefit from as much feedback as possible.

Cheers,

Till

On Tue, Mar 17, 2020 at 2:57 AM LakeShen <[hidden email]> wrote:

Wow,this feature is look so good,I will see this feature.

Thank you to reply me , Till😀.

Best Regards,

LakeShen

Till Rohrmann <[hidden email]> 于2020年3月17日周二 上午4:17写道：

If you want to change the max parallelism then you need to take a savepoint and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html

Cheers,

Till

On Sat, Mar 14, 2020 at 4:07 AM LakeShen <[hidden email]> wrote:

Hi Eleanore , if you resume from savepoint , you can't change the flink
operator's max parallelism .

Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道：

> Hi Piotr,
> Does this also apply to savepoint? (meaning the max parallelism should not
> change for job resume from savepoint?)
>
> Thanks a lot!
> Eleanore
>
> On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Yes, you can change the parallelism. One thing that you can not change is
> > “max parallelism”.
> >
> > Piotrek
> >
> > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]>
> > wrote:
> > >
> > > I think you can modify the operator’s parallelism. It is only if you
> > have set maxParallelism, and while restoring from a checkpoint, you
> > shouldn’t modify the maxParallelism. Otherwise, I believe the state will
> be
> > lost.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]
> > <mailto:[hidden email]>> wrote:
> > > Hi community,
> > >       I have a question is that I cancel the flink task and retain the
> > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > flink operator's parallelism,in my thoughts, I think I can't change the
> > flink operator's parallelism,but I am not sure.
> > >      Thanks to your reply.
> > >
> > > Best wishes,
> > > LakeShen
> >
> >
>

[jinhai wang]

Hi LakeShen

You also must assign IDs to all operators of an application. Otherwise, you may not be able to recover from checkpoint

Doc: https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best Regards

[hidden email]

2020年3月17日 下午5:51，Till Rohrmann <[hidden email]> 写道：

Let us know if you should run into any problems. The state processor API is still young and could benefit from as much feedback as possible.

Cheers,

Till

On Tue, Mar 17, 2020 at 2:57 AM LakeShen <[hidden email]> wrote:

Wow,this feature is look so good,I will see this feature.

Thank you to reply me , Till😀.

Best Regards,

LakeShen

Till Rohrmann <[hidden email]> 于2020年3月17日周二 上午4:17写道：

If you want to change the max parallelism then you need to take a savepoint and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html

Cheers,

Till

On Sat, Mar 14, 2020 at 4:07 AM LakeShen <[hidden email]> wrote:

Hi Eleanore , if you resume from savepoint , you can't change the flink
operator's max parallelism .

Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道：

> Hi Piotr,
> Does this also apply to savepoint? (meaning the max parallelism should not
> change for job resume from savepoint?)
>
> Thanks a lot!
> Eleanore
>
> On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Yes, you can change the parallelism. One thing that you can not change is
> > “max parallelism”.
> >
> > Piotrek
> >
> > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]>
> > wrote:
> > >
> > > I think you can modify the operator’s parallelism. It is only if you
> > have set maxParallelism, and while restoring from a checkpoint, you
> > shouldn’t modify the maxParallelism. Otherwise, I believe the state will
> be
> > lost.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]
> > <mailto:[hidden email]>> wrote:
> > > Hi community,
> > >       I have a question is that I cancel the flink task and retain the
> > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > flink operator's parallelism,in my thoughts, I think I can't change the
> > flink operator's parallelism,but I am not sure.
> > >      Thanks to your reply.
> > >
> > > Best wishes,
> > > LakeShen
> >
> >
>

[LakeShen]

Thank you, I will do that. 

jinhai wang <[hidden email]> 于2020年3月17日周二 下午5:58写道：

Hi LakeShen

You also must assign IDs to all operators of an application. Otherwise, you may not be able to recover from checkpoint

Doc: https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best Regards

[hidden email]

2020年3月17日 下午5:51，Till Rohrmann <[hidden email]> 写道：

Let us know if you should run into any problems. The state processor API is still young and could benefit from as much feedback as possible.

Cheers,

Till

On Tue, Mar 17, 2020 at 2:57 AM LakeShen <[hidden email]> wrote:

Wow,this feature is look so good,I will see this feature.

Thank you to reply me , Till😀.

Best Regards,

LakeShen

Till Rohrmann <[hidden email]> 于2020年3月17日周二 上午4:17写道：

If you want to change the max parallelism then you need to take a savepoint and use Flink's state processor API [1] to rewrite the max parallelism by creating a new savepoint from the old one.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html

Cheers,

Till

On Sat, Mar 14, 2020 at 4:07 AM LakeShen <[hidden email]> wrote:

Hi Eleanore , if you resume from savepoint , you can't change the flink
operator's max parallelism .

Eleanore Jin <[hidden email]> 于2020年3月14日周六 上午12:51写道：

> Hi Piotr,
> Does this also apply to savepoint? (meaning the max parallelism should not
> change for job resume from savepoint?)
>
> Thanks a lot!
> Eleanore
>
> On Fri, Mar 13, 2020 at 6:33 AM Piotr Nowojski <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Yes, you can change the parallelism. One thing that you can not change is
> > “max parallelism”.
> >
> > Piotrek
> >
> > > On 13 Mar 2020, at 04:34, Sivaprasanna <[hidden email]>
> > wrote:
> > >
> > > I think you can modify the operator’s parallelism. It is only if you
> > have set maxParallelism, and while restoring from a checkpoint, you
> > shouldn’t modify the maxParallelism. Otherwise, I believe the state will
> be
> > lost.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Fri, 13 Mar 2020 at 9:01 AM, LakeShen <[hidden email]
> > <mailto:[hidden email]>> wrote:
> > > Hi community,
> > >       I have a question is that I cancel the flink task and retain the
> > checkpoint dir, then restore from the checkpoint dir ,can I change the
> > flink operator's parallelism,in my thoughts, I think I can't change the
> > flink operator's parallelism,but I am not sure.
> > >      Thanks to your reply.
> > >
> > > Best wishes,
> > > LakeShen
> >
> >
>