State migration into multiple operators

[bastien dine]

Hello,

I would like to have some advices about splitting an operator with a state into multiple operators.

The new operators would have state containing pieces of information of the initial state

We will "split" the state

For exemple, I have operator (process) with uid A, with a state containing field1, field2

I would like to split it into two operator B & C with, respectively, state field1 and state field2

How can I split my state upon multiple operators ?

Best Regards,

Bastien 

[Piotr Nowojski-3]

Hi,

Currently there is no native Flink support for modifying the state in a such manner. However there is an on-going effort [1] and a third party project [2] to address exactly this. Both allows you you to read savepoint, modify it and write back the new modified savepoint from which you can restore.

[1] https://github.com/sjwiesman/flink/commits/savepoint-connector

[2] https://github.com/king/bravo

You can take a look at them.

Piotrek

On 14 May 2019, at 09:55, bastien dine <[hidden email]> wrote:

Hello,

I would like to have some advices about splitting an operator with a state into multiple operators.

The new operators would have state containing pieces of information of the initial state

We will "split" the state

For exemple, I have operator (process) with uid A, with a state containing field1, field2

I would like to split it into two operator B & C with, respectively, state field1 and state field2

How can I split my state upon multiple operators ?

Best Regards,

Bastien 

[Tzu-Li (Gordon) Tai]

Hi,

Just to add to what Piotr already mentioned:
The community is working on adding support for this directly in Flink.
You can follow the efforts here: https://issues.apache.org/jira/browse/FLINK-12047.

Cheers,
Gordon

On Tue, May 14, 2019 at 11:39 AM Piotr Nowojski <[hidden email]> wrote:

Hi,

Currently there is no native Flink support for modifying the state in a such manner. However there is an on-going effort [1] and a third party project [2] to address exactly this. Both allows you you to read savepoint, modify it and write back the new modified savepoint from which you can restore.

[1] https://github.com/sjwiesman/flink/commits/savepoint-connector

[2] https://github.com/king/bravo

You can take a look at them.

Piotrek

On 14 May 2019, at 09:55, bastien dine <[hidden email]> wrote:

Hello,

I would like to have some advices about splitting an operator with a state into multiple operators.

The new operators would have state containing pieces of information of the initial state

We will "split" the state

For exemple, I have operator (process) with uid A, with a state containing field1, field2

I would like to split it into two operator B & C with, respectively, state field1 and state field2

How can I split my state upon multiple operators ?

Best Regards,

Bastien