State expiration in Flink

[Vasily Melnik]

Hi .
I'm a bit confused with this point in State TTL documentation:
" By default, expired values are explicitly removed on read, such as ValueState#value, and periodically garbage collected in the background if supported by the configured state backend. "
Does it mean, that  if i have only one event with specific key, it's state will never be cleaned on TTL expiration cause of i will never call value method for this key again?

[Yun Tang]

Hi Vasily

After Flink-1.10, state will be cleaned up periodically as CleanupInBackground is enabled by default. Thus, even you never access some specific entry of state and that entry could still be cleaned up.

Best

Yun Tang

---

From: Vasily Melnik <[hidden email]>
Sent: Saturday, May 30, 2020 23:29
To: user <[hidden email]>
Subject: State expiration in Flink

 

Hi .
I'm a bit confused with this point in State TTL documentation:
" By default, expired values are explicitly removed on read, such as ValueState#value, and periodically garbage collected in the background if supported by the configured state backend. "
Does it mean, that  if i have only one event with specific key, it's state will never be cleaned on TTL expiration cause of i will never call value method for this key again?

[Vasily Melnik]

Thanks, Yun!

One more question: is it possible to create some kind of handler on clearing up the state? For example i want to flush state to external storage (e.g. HBase) before cleanup.Now we make this manually with onTimer method, but is there another way?

On Mon, 1 Jun 2020 at 05:28, Yun Tang <[hidden email]> wrote:

Hi Vasily

After Flink-1.10, state will be cleaned up periodically as CleanupInBackground is enabled by default. Thus, even you never access some specific entry of state and that entry could still be cleaned up.

Best

Yun Tang

---

From: Vasily Melnik <[hidden email]>
Sent: Saturday, May 30, 2020 23:29
To: user <[hidden email]>
Subject: State expiration in Flink

 

Hi .
I'm a bit confused with this point in State TTL documentation:
" By default, expired values are explicitly removed on read, such as ValueState#value, and periodically garbage collected in the background if supported by the configured state backend. "
Does it mean, that  if i have only one event with specific key, it's state will never be cleaned on TTL expiration cause of i will never call value method for this key again?

[Yun Tang]

Hi Vasily

As far as I know, current TTL of state lack of such kind of trigger, and perhaps onTimer or process specific event to trigger could help your scenario.

Best

Yun Tang.

---

From: Vasily Melnik <[hidden email]>
Sent: Monday, June 1, 2020 14:13
To: Yun Tang <[hidden email]>
Cc: user <[hidden email]>
Subject: Re: State expiration in Flink

 

Thanks, Yun!

One more question: is it possible to create some kind of handler on clearing up the state? For example i want to flush state to external storage (e.g. HBase) before cleanup.Now we make this manually with onTimer method, but is there another way?

On Mon, 1 Jun 2020 at 05:28, Yun Tang <[hidden email]> wrote:

Hi Vasily

After Flink-1.10, state will be cleaned up periodically as CleanupInBackground is enabled by default. Thus, even you never access some specific entry of state and that entry could still be cleaned up.

Best

Yun Tang

---

From: Vasily Melnik <[hidden email]>
Sent: Saturday, May 30, 2020 23:29
To: user <[hidden email]>
Subject: State expiration in Flink

 

Hi .
I'm a bit confused with this point in State TTL documentation:
" By default, expired values are explicitly removed on read, such as ValueState#value, and periodically garbage collected in the background if supported by the configured state backend. "
Does it mean, that  if i have only one event with specific key, it's state will never be cleaned on TTL expiration cause of i will never call value method for this key again?