hybrid state backends

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

hybrid state backends

Marco Villalobos-2
Is it possible to use different statebackends for different operators? There are certain situations where I want the state to reside completely in memory, and other situations where I want it stored in rocksdb.
Reply | Threaded
Open this post in threaded view
|

Re: hybrid state backends

Yun Gao

Hi Marco,

Sorry that current statebackend is a global configuration and could 
not be configured differently for different operators.

One possible alternative option to this requirements might be set rocksdb 
as the default statebackend, and for those operators that want to put state
in memory, a new operator might be implemented by extends AbstractStreamOperator
and rewrite the snapshotState() method, and use raw state to snapshot the in-memory
data. However, this option would touch some non-user-level api of flink.

Best,
 Yun
------------------------------------------------------------------
Sender:Marco Villalobos<[hidden email]>
Date:2021/02/05 19:09:37
Recipient:user<[hidden email]>
Theme:hybrid state backends

Is it possible to use different statebackends for different operators? There are certain situations where I want the state to reside completely in memory, and other situations where I want it stored in rocksdb.