State backend questions

[Christophe Jolif]

Hi all,

At first my state should not be "that" big and fit in memory, so FsStateBackend could be a solution for me. However moving forward I envision more features and more users and the state growing. With that in mind RocksDBStateBackend might be the solution.

Is there an easy "upgrade" path from one to another? In other words is there an easy path to "move" the state from one backend to another one and restart the job from there if the need arise or should I definitely plan long ahead and use RocksDB right away if I don't want to get into trouble?

If yes, how much configuration is available at Flink level to configure memory used by RocksDB to cache data without having to go back to disk so that I don't penalize too much the current use-cases?

Thanks,

--

[Chesnay Schepler]

According to this thread it is not yet possible to switch to/from RocksDBStatebackend, so I would suggest to with RocksDB from the start.

For tuning RocksDB, see here.

On 17.01.2018 01:30, Christophe Jolif wrote:

Hi all,

At first my state should not be "that" big and fit in memory, so FsStateBackend could be a solution for me. However moving forward I envision more features and more users and the state growing. With that in mind RocksDBStateBackend might be the solution.

Is there an easy "upgrade" path from one to another? In other words is there an easy path to "move" the state from one backend to another one and restart the job from there if the need arise or should I definitely plan long ahead and use RocksDB right away if I don't want to get into trouble?

If yes, how much configuration is available at Flink level to configure memory used by RocksDB to cache data without having to go back to disk so that I don't penalize too much the current use-cases?

Thanks,

--