Flink State Processor API - Bootstrap One state

[ApoorvK]

Currently my flink application has state size of 160GB(around 50 operators),
where few state operator size is much higher, I am planning to use state
processor API to bootstrap let say one particular state having operator id
o1 and inside is a ValueState s1 as ID.

Following steps I have planned to do it :

1. If I take a savepoint of the application I will be having a state folder
containing operator and meta data file having o1 and s1.

2. Now II read only that state using state processor API (another flink
app)and re-write it with the data that I want with same o1 and s1 ids and
copy paste this folder to the savepoint folder taken in step 1.

3. Restore the application with the savepoint taken in step 1.

Doing so as I do not want to touch any other state , I have my concerned
with a particular state operator.

Team, Kindly let me know if this is the right way to do it, or is there any
better way using which I can achieve this.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Tzu-Li (Gordon) Tai]

Hi,

Using the State Processor API, modifying the state in an existing savepoint results in a new savepoint (new directory) with the new modified state.
The original savepoint remains intact.
The API allows you to only touch certain operators, without having to touch any other state and have them remain as is.

Note that when generating a new savepoint from an existing savepoint, the State Processor API does not perform a deep copy for untouched state.
Instead, the new savepoint will contain references to the old savepoint for these untouched state.
Essentially these means that modified savepoints written by the State Processor API are not self-contained, and you should be careful not to delete the original savepoints as that would invalidate the generated new one.

Cheers,
Gordon

On Tue, Nov 17, 2020 at 2:19 PM ApoorvK <[hidden email]> wrote:

Currently my flink application has state size of 160GB(around 50 operators),
where few state operator size is much higher, I am planning to use state
processor API to bootstrap let say one particular state having operator id
o1 and inside is a ValueState s1 as ID.

Following steps I have planned to do it :

1. If I take a savepoint of the application I will be having a state folder
containing operator and meta data file having o1 and s1.

2. Now II read only that state using state processor API (another flink
app)and re-write it with the data that I want with same o1 and s1 ids and
copy paste this folder to the savepoint folder taken in step 1.

3. Restore the application with the savepoint taken in step 1.

Doing so as I do not want to touch any other state , I have my concerned
with a particular state operator.

Team, Kindly let me know if this is the right way to do it, or is there any
better way using which I can achieve this.



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/