TTL state migration

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

TTL state migration

Ning Shi
I have a job using TTL map state and RocksDB state backend. If I
lengthen the TTL on the map state and resume the job from savepoint (or
checkpoint assuming I don't change the state backend), will new values
added to that map have the new TTL or will the old TTL in the savepoint
override my changes?

Thanks,

Ning
Reply | Threaded
Open this post in threaded view
|

Re: TTL state migration

Andrey Zagrebin
Hi Ning,

State backends store the timestamp of last access/modification of state with TTL.
This absolute timestamp does not depend on the configured time-to-live.
If you restart a job from the savepoint and configure longer time-to-live than before,
it just means that map state entries, which have not been cleaned up before savepoint,
and newly added entries will expire later relative to their originally saved access timestamp.

Best,
Andrey

> On 5 Dec 2018, at 04:20, Ning Shi <[hidden email]> wrote:
>
> I have a job using TTL map state and RocksDB state backend. If I
> lengthen the TTL on the map state and resume the job from savepoint (or
> checkpoint assuming I don't change the state backend), will new values
> added to that map have the new TTL or will the old TTL in the savepoint
> override my changes?
>
> Thanks,
>
> Ning

Reply | Threaded
Open this post in threaded view
|

Re: TTL state migration

Ning Shi
Hi Andrey,

Thank you. That makes sense. Looking at the code of how TTL is
implemented, it matches exactly what you said.

Ning

On Wed, Dec 5, 2018 at 9:37 AM Andrey Zagrebin <[hidden email]> wrote:

>
> Hi Ning,
>
> State backends store the timestamp of last access/modification of state with TTL.
> This absolute timestamp does not depend on the configured time-to-live.
> If you restart a job from the savepoint and configure longer time-to-live than before,
> it just means that map state entries, which have not been cleaned up before savepoint,
> and newly added entries will expire later relative to their originally saved access timestamp.
>
> Best,
> Andrey
>
> > On 5 Dec 2018, at 04:20, Ning Shi <[hidden email]> wrote:
> >
> > I have a job using TTL map state and RocksDB state backend. If I
> > lengthen the TTL on the map state and resume the job from savepoint (or
> > checkpoint assuming I don't change the state backend), will new values
> > added to that map have the new TTL or will the old TTL in the savepoint
> > override my changes?
> >
> > Thanks,
> >
> > Ning
>