FileSystem vs RocksDb backend

[Fritz Budiyanto]

Hi All,

If my states fit in the JVM heap, should I be using filesystem backend with HDFS instead of RocksDB ?

Could someone comment on the FS vs RocksDb backend performance ?

I’ve also heard that RocksDb backend is required to support delta checkpointing. Is this true ?
Is delta checkpointing part of 1.3.0 ?


Thanks,
Fritz

[Fabian Hueske-2]

Hi Fritz,

the FSStateBackend is a lot faster than the RocksDBStateBackend.
I don't know exact numbers but if I remember correctly somebody mentioned a 10x difference.

Delta checkpointing is only supported by RocksDB since Flink 1.3.
Given that the FSStateBackend is only suitable for small states, this should not be significant limitation.

Best, Fabian

2017-06-02 19:20 GMT+02:00 Fritz Budiyanto <[hidden email]>:

Hi All,

If my states fit in the JVM heap, should I be using filesystem backend with HDFS instead of RocksDB ?

Could someone comment on the FS vs RocksDb backend performance ?

I’ve also heard that RocksDb backend is required to support delta checkpointing. Is this true ?
Is delta checkpointing part of 1.3.0 ?


Thanks,
Fritz