RocksDB Number of Keys Metric

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

RocksDB Number of Keys Metric

Ahmed
Is there a clean way of exposing a metrics regarding the number of keys (even if it is an estimate using 'rocksdb.estimate-num-keys') in a rocksdb state store? RocksDBValueState is not public to users code.

Best,
Ahmed

Reply | Threaded
Open this post in threaded view
|

Re: RocksDB Number of Keys Metric

vino yang
Hi Ahmed,

If you feel that this metric is necessary, you can create an issue in JIRA, then the problem may be more easily seen by the relevant people. 
If you need to answer this question, maybe it is more effective to ping @Andrey?

Thanks, vino.

Ahmed <[hidden email]> 于2018年9月2日周日 上午2:31写道:
Is there a clean way of exposing a metrics regarding the number of keys (even if it is an estimate using 'rocksdb.estimate-num-keys') in a rocksdb state store? RocksDBValueState is not public to users code.

Best,
Ahmed

Reply | Threaded
Open this post in threaded view
|

Re: RocksDB Number of Keys Metric

Andrey Zagrebin
Hi,

afaik, the option is not exposed according to the current state of source code.
I can see it to be useful and technically possible using:
db.getLongProperty(stateColumnFamilyHandle, "rocksdb.estimate-num-keys”);

Though couple of things come into my mind to take into account for this feature:

- we can expose it per state object for heap and rocksdb, e.g. as State.estimatekeyNum() 
  but it will report it only for key groups in the operator subtask where it is called.

- another way might be to expose it in Web UI where we can even sum it up for all operator subtasks.

- for all state types currently, "rocksdb.estimate-num-keys” would show total number of all state keys but
  for map state this particular metric would show total number of user map keys for all state
  keys, so it might be misleading and technically not so easy to align its semantics for map with other state types.

We can discuss here in mailing list, how many users would be interested in this feature and
then a JIRA ticket can be created to track it, if community decides to contribute this feature.

Cheers,
Andrey

On 4 Sep 2018, at 05:49, vino yang <[hidden email]> wrote:

Hi Ahmed,

If you feel that this metric is necessary, you can create an issue in JIRA, then the problem may be more easily seen by the relevant people. 
If you need to answer this question, maybe it is more effective to ping @Andrey?

Thanks, vino.

Ahmed <[hidden email]> 于2018年9月2日周日 上午2:31写道:
Is there a clean way of exposing a metrics regarding the number of keys (even if it is an estimate using 'rocksdb.estimate-num-keys') in a rocksdb state store? RocksDBValueState is not public to users code.

Best,
Ahmed