ListState with millions of elements

[Aaron Levin]

Hello friendly Flink community!

I'm curious if anyone has operational experience with jobs that store ListState where occasionally, due to skew, some small number of lists stored in ListState (stored in RocksDB) will have millions of elements. Here are the stats:

* millions of keys

* p95 size of list in ListState is ~2.

* some small number of keys (less than 100) may have lists whose size is on the order of tens of thousands and up to millions.

* state is stored in RocksDB

Are there any known issues or limitations with storing or fetching that much list state out of RocksDB? I realize fetching from RocksDB and deserializing will be costly when hitting a key with a list of a million elements, but is there anything else we should consider?

Thanks!

Best,

Aaron Levin

[Seth Wiesman]

There is a limitation in RocksDB's JNI bridge that will cause applications to fail if list state exceeds 2GB. I am not aware of anyone working on this issue.

Seth.

[1] https://github.com/facebook/rocksdb/issues/2383

On Wed, Apr 8, 2020 at 12:02 PM Aaron Levin <[hidden email]> wrote:

Hello friendly Flink community!

I'm curious if anyone has operational experience with jobs that store ListState where occasionally, due to skew, some small number of lists stored in ListState (stored in RocksDB) will have millions of elements. Here are the stats:

* millions of keys

* p95 size of list in ListState is ~2.

* some small number of keys (less than 100) may have lists whose size is on the order of tens of thousands and up to millions.

* state is stored in RocksDB

Are there any known issues or limitations with storing or fetching that much list state out of RocksDB? I realize fetching from RocksDB and deserializing will be costly when hitting a key with a list of a million elements, but is there anything else we should consider?

Thanks!

Best,

Aaron Levin

[Aljoscha Krettek]

On 08.04.20 20:14, Seth Wiesman wrote:
> There is a limitation in RocksDB's JNI bridge that will cause applications
> to fail if list state exceeds 2GB. I am not aware of anyone working on this
> issue.

That is correct, and here's the Jira issue for it:
https://issues.apache.org/jira/browse/FLINK-6761

There's also another issue about changing RocksDB list state to not
store the whole ListState in a single RocksDB value but instead store
the individual ListState entries in individual RocksDB values, under a
specially formed "sequential" key. This is the issue:
https://issues.apache.org/jira/browse/FLINK-8297

Best,
Aljoscha