Are Dynamic tables backed by rocksdb?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Are Dynamic tables backed by rocksdb?

kant kodali
Hi All,

Are Dynamic tables backed by Rocksdb or in memory? if they are backed by RocksDB can I use SQL to query the state?

Thanks! 
Reply | Threaded
Open this post in threaded view
|

Re: Are Dynamic tables backed by rocksdb?

Fabian Hueske-2
Hi,

Dynamic tables might not be persisted at all but only when it is necessary for the computation of a query.
For example a simple "SELECT * FROM t WHERE a = 1" query on an append only table t does not require to persist t.

However, there are a bunch of operations that require to store some parts of a dynamic table but not necessarily the full table.
All operators that need to store data put it into regular Flink state. This means that all state is stored in the configured state backend.
You cannot directly query the data from there, but could use the state processor API. However, this also assumes that you know the internal data representation that the operators use.

Best, Fabian


Am Do., 31. Okt. 2019 um 09:51 Uhr schrieb kant kodali <[hidden email]>:
Hi All,

Are Dynamic tables backed by Rocksdb or in memory? if they are backed by RocksDB can I use SQL to query the state?

Thanks!