Re: Migrating Existing TTL State to 1.8
Posted by
Stefan Richter-4 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Migrating-Existing-TTL-State-to-1-8-tp26622p26688.html
Hi,
If you are worried about old state, you can combine the compaction filter based TTL with other cleanup strategies (see docs). For example, setting `cleanupFullSnapshot` when you take a savepoint it will be cleared of any expired state and you can then use it to bring it into Flink 1.8.
Best,
Stefan
Just wondering if anyone has any insights into the new TTL state cleanup feature mentioned below.
Thanks,
It's exciting to see TTL state cleanup feature in 1.8. I have a question regarding the migration of existing TTL state to the newer version.
Looking at the Pull Request [1] that introduced this feature, it seems like that Flink is leveraging RocksDB's compaction filter to remove stale state. I assume this means that state will only be cleaned on compaction. If I have a significant amount of stale TTL state, some of which may have already been compacted to higher levels already, upgrading to 1.8 may not clean them. Is this assumption correct? If so, is the best approach to take a full snapshot/checkpoint and restore it to 1.8 to have them been cleaned on initialization?
Thanks,
--