Reduce key state

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Reduce key state

Frank Wilson
Hi,

In an unwindowed key stream while using event time semantics is state stored indefinitely or does it get expired eventually (was wondering if the state inherits the event time of the element that updated, and if it expires when the watermark goes past it).

Thanks,

Frank
Reply | Threaded
Open this post in threaded view
|

Re: Reduce key state

Fabian Hueske-2
Hi Frank,

By default, Flink does not remove any state. It is the responsibility of the developer to ensure that an application does not leak state.
Typically, you would use timers [1] to discard state that expired and is not useful anymore.

In the last release 1.8, we added lazy cleanup strategies for State TTL (time-to-live) [2] [3].
You configure a time-to-live when you define state and state is removed if it wasn't touched for the configured interval of time. There are a bunch of config options (is read and/or write considered as touch, strict read protection, etc.). Right now, only processing time is supported.

Best, Fabian


Am Do., 9. Mai 2019 um 15:01 Uhr schrieb Frank Wilson <[hidden email]>:
Hi,

In an unwindowed key stream while using event time semantics is state stored indefinitely or does it get expired eventually (was wondering if the state inherits the event time of the element that updated, and if it expires when the watermark goes past it).

Thanks,

Frank