Is State TTL possible with event-time characteristics ?

[parti]

With Flink 1.9 is state TTL supported for event-time characteristics? This part of the documentation says that 

• Only TTLs in reference to processing time are currently supported.

Does this mean if a program uses event-time characteristics with stateful operators, it can not use TTL ??

Also clean up section of the documentation says state values that are never read will never be cleared.

The question is, when processing a stream with unique elements or keys why would Flink framework expect the same key to be read in order for it to be removed after its expiration time ? Why does it not simply clean up the value for that key based on timers automatically without waiting for read operation from user code?

Thanks 

Arti

[Congxian Qiu]

Hi

 Currently, Flink does not support event-time TTL state, there is an issue[1] tracking this. 

[1] https://issues.apache.org/jira/browse/FLINK-12005

Arti Pande <[hidden email]> 于2020年6月17日周三 下午7:37写道：

With Flink 1.9 is state TTL supported for event-time characteristics? This part of the documentation says that 

• Only TTLs in reference to processing time are currently supported.

Does this mean if a program uses event-time characteristics with stateful operators, it can not use TTL ??

Also clean up section of the documentation says state values that are never read will never be cleared.

The question is, when processing a stream with unique elements or keys why would Flink framework expect the same key to be read in order for it to be removed after its expiration time ? Why does it not simply clean up the value for that key based on timers automatically without waiting for read operation from user code?

Thanks 

Arti

[Andrey Zagrebin-5]

Hi Arti,

Any program can use State with TTL but the state can only expire in processing time at the moment even if you configure event-time characteristics.

As Congxian mentioned, the event time for TTL is planned.

The cleanup says that it will not be removed 'by default'. The following sections [1] describe background cleanup which is not activated 'by default' in 1.9 but in 1.10.

If you activate the background cleanup, you do not have to read the expired state to clean it up as if you have those timers you mentioned.

See also the docs for details about background cleanup caveats.

The timers approach is a valid way but heavy-weight in terms of storage because Flink will have to create a separate state for timers: key/timestamp.

The timers approach is not implemented in Flink out-of-the-box at the moment. It can be implemented in the application as a simple background cleanup.

Best,

Andrey

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/state/state.html#cleanup-in-background

On Wed, Jun 17, 2020 at 3:07 PM Congxian Qiu <[hidden email]> wrote:

Hi

 Currently, Flink does not support event-time TTL state, there is an issue[1] tracking this. 

[1] https://issues.apache.org/jira/browse/FLINK-12005

Best,

Congxian

Arti Pande <[hidden email]> 于2020年6月17日周三 下午7:37写道：

With Flink 1.9 is state TTL supported for event-time characteristics? This part of the documentation says that 

• Only TTLs in reference to processing time are currently supported.

Does this mean if a program uses event-time characteristics with stateful operators, it can not use TTL ??

Also clean up section of the documentation says state values that are never read will never be cleared.

The question is, when processing a stream with unique elements or keys why would Flink framework expect the same key to be read in order for it to be removed after its expiration time ? Why does it not simply clean up the value for that key based on timers automatically without waiting for read operation from user code?

Thanks 

Arti