Dear Flink users and developers!
I start this discussion to collect feedback about maintaining a custom RocksDB branch for Flink, if anyone sees any problems with this approach. Are there people who already uses a custom RocksDB client build with RocksDB state backend?
As you might already know, the community is trying currently to make some improvements in RocksDB state backend which require a newer version of RocksDB client and some Flink specific changes of its code base.
In particular:
- custom c++ compaction filter for background cleanup of expired state entries with TTL [1]
- merge operator without separating commas for the list state [2]
The TTL compaction filter is too Flink specific to merge it into the RocksDB repository.
The second one might take time to be merged and released in RocksDB.
The plug-ability of filter/merge operators in RocksDB would allow the code to be kept on Flink side but it is currently under development and ETA is not cleat at the moment.
To unblock releasing of related Flink features, we suggest to create and maintain a branch of RocksDB which would include these changes, as it happened already in the past. The RocksDB backend module can be built against this branch. We can get back to normal RocksDB releases, once it allows Flink code to be kept out of it.
Thanks,
Andrey
Locked
