Need a clarification about removing a stateful operator

[Tony Wei]

Hi all,

I'm confused about the description in documentation. [1]

• Removing a stateful operator: The state of the removed operator is lost unless
  another operator takes it over. When starting the upgraded application, you have
  to explicitly agree to discard the state.

Does that mean if I take a full snapshot (e.g. savepoint) after restoring by explicitly agreeing to

discard the state, then the state won't exist in that snapshot? Or does it just mean ignore the

state but the state still exist forever, unless I explicitly purge that state by using state operator?

And could this behavior differ between different state backend (Memory, FS, RocksDB) ?

Many thanks,

Tony Wei

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology

[Chesnay Schepler]

The state won't exist in the snapshot.

On 17.08.2018 04:38, Tony Wei wrote:

Hi all,

I'm confused about the description in documentation. [1]

• Removing a stateful operator: The state of the removed operator is lost unless
  another operator takes it over. When starting the upgraded application, you have
  to explicitly agree to discard the state.

Does that mean if I take a full snapshot (e.g. savepoint) after restoring by explicitly agreeing to

discard the state, then the state won't exist in that snapshot? Or does it just mean ignore the

state but the state still exist forever, unless I explicitly purge that state by using state operator?

And could this behavior differ between different state backend (Memory, FS, RocksDB) ?

Many thanks,

Tony Wei

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology

[Tony Wei]

Hi Chesnay,

Thanks for your quick reply. I have another question. Will the state, which is ignored, be transported

to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and only those states reuired

by operators be transported to each TM?

Best,

Tony Wei

2018-08-17 14:38 GMT+08:00 Chesnay Schepler <[hidden email]>:

The state won't exist in the snapshot.

On 17.08.2018 04:38, Tony Wei wrote:

Hi all,

I'm confused about the description in documentation. [1]

• Removing a stateful operator: The state of the removed operator is lost unless
  another operator takes it over. When starting the upgraded application, you have
  to explicitly agree to discard the state.

Does that mean if I take a full snapshot (e.g. savepoint) after restoring by explicitly agreeing to

discard the state, then the state won't exist in that snapshot? Or does it just mean ignore the

state but the state still exist forever, unless I explicitly purge that state by using state operator?

And could this behavior differ between different state backend (Memory, FS, RocksDB) ?

Many thanks,

Tony Wei

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology

[Stefan Richter]

Hi,

it will not be transported. The JM does the state assignment to create the deployment information for all tasks. If will just exclude the state for operators that are not present. So in your next checkpoints they will no longer be contained.

Best,

Stefan

Am 17.08.2018 um 09:26 schrieb Tony Wei <[hidden email]>:

Hi Chesnay,

Thanks for your quick reply. I have another question. Will the state, which is ignored, be transported

to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and only those states reuired

by operators be transported to each TM?

Best,

Tony Wei

2018-08-17 14:38 GMT+08:00 Chesnay Schepler <[hidden email]>:

The state won't exist in the snapshot.

On 17.08.2018 04:38, Tony Wei wrote:

Hi all,

I'm confused about the description in documentation. [1]

• Removing a stateful operator: The state of the removed operator is lost unless
  another operator takes it over. When starting the upgraded application, you have
  to explicitly agree to discard the state.

Does that mean if I take a full snapshot (e.g. savepoint) after restoring by explicitly agreeing to

discard the state, then the state won't exist in that snapshot? Or does it just mean ignore the

state but the state still exist forever, unless I explicitly purge that state by using state operator?

And could this behavior differ between different state backend (Memory, FS, RocksDB) ?

Many thanks,

Tony Wei

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology

[Tony Wei]

Hi Stefan,

Thanks for your detailed explanation.

Best,

Tony Wei

2018-08-17 15:56 GMT+08:00 Stefan Richter <[hidden email]>:

Hi,

it will not be transported. The JM does the state assignment to create the deployment information for all tasks. If will just exclude the state for operators that are not present. So in your next checkpoints they will no longer be contained.

Best,

Stefan

Am 17.08.2018 um 09:26 schrieb Tony Wei <[hidden email]>:

Hi Chesnay,

Thanks for your quick reply. I have another question. Will the state, which is ignored, be transported

to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and only those states reuired

by operators be transported to each TM?

Best,

Tony Wei

2018-08-17 14:38 GMT+08:00 Chesnay Schepler <[hidden email]>:

The state won't exist in the snapshot.

On 17.08.2018 04:38, Tony Wei wrote:

Hi all,

I'm confused about the description in documentation. [1]

• Removing a stateful operator: The state of the removed operator is lost unless
  another operator takes it over. When starting the upgraded application, you have
  to explicitly agree to discard the state.

Does that mean if I take a full snapshot (e.g. savepoint) after restoring by explicitly agreeing to

discard the state, then the state won't exist in that snapshot? Or does it just mean ignore the

state but the state still exist forever, unless I explicitly purge that state by using state operator?

And could this behavior differ between different state backend (Memory, FS, RocksDB) ?

Many thanks,

Tony Wei

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology