Rocksdb Metrics

[Sayat Satybaldiyev]

Flink provides a rich number of metrics. However, I didn't find any metrics for rocksdb state backend not in metrics doc nor in JMX Mbean. 

Is there are any metrics for the rocksdb backend that Flink exposes?

[Stefan Richter]

Hi,

this feature is tracked here https://issues.apache.org/jira/browse/FLINK-10423

Best,

Stefan

Am 25.09.2018 um 17:51 schrieb Sayat Satybaldiyev <[hidden email]>:

Flink provides a rich number of metrics. However, I didn't find any metrics for rocksdb state backend not in metrics doc nor in JMX Mbean. 

Is there are any metrics for the rocksdb backend that Flink exposes?

[Yun Tang]

Hi Sayat

Before this future is on, you could also find some metrics information, such as hit/miss count, file status from RocksDB itself. By default, RocksDB will dump its stats to its information LOG file every 10 minutes (you could call DBOptions.setStatsDumpPeriodSec to reduce the time interval), and you could find the information LOG file under rocksDB state-backend's db folder.

Best

Yun

---

From: Stefan Richter <[hidden email]>
Sent: Wednesday, September 26, 2018 0:56
To: Sayat Satybaldiyev
Cc: [hidden email]
Subject: Re: Rocksdb Metrics

 

Hi,

this feature is tracked here https://issues.apache.org/jira/browse/FLINK-10423

Best,

Stefan

Am 25.09.2018 um 17:51 schrieb Sayat Satybaldiyev <[hidden email]>:

Flink provides a rich number of metrics. However, I didn't find any metrics for rocksdb state backend not in metrics doc nor in JMX Mbean. 

Is there are any metrics for the rocksdb backend that Flink exposes?

[Sayat Satybaldiyev]

Thank you for this information. @Yun is there an easy way to expose a number of records in rockdsdb?

On Wed, Sep 26, 2018 at 9:47 AM Yun Tang <[hidden email]> wrote:

Hi Sayat

Before this future is on, you could also find some metrics information, such as hit/miss count, file status from RocksDB itself. By default, RocksDB will dump its stats to its information LOG file every 10 minutes (you could call DBOptions.setStatsDumpPeriodSec to reduce the time interval), and you could find the information LOG file under rocksDB state-backend's db folder.

Best

Yun

---

From: Stefan Richter <[hidden email]>
Sent: Wednesday, September 26, 2018 0:56
To: Sayat Satybaldiyev
Cc: [hidden email]
Subject: Re: Rocksdb Metrics

 

Hi,

this feature is tracked here https://issues.apache.org/jira/browse/FLINK-10423

Best,

Stefan

Am 25.09.2018 um 17:51 schrieb Sayat Satybaldiyev <[hidden email]>:

Flink provides a rich number of metrics. However, I didn't find any metrics for rocksdb state backend not in metrics doc nor in JMX Mbean. 

Is there are any metrics for the rocksdb backend that Flink exposes?

[Sayat Satybaldiyev]

actually, once I wrote my question I've realized that I can do it with custom metrics and getting easily the size of the state map.

On Wed, Sep 26, 2018 at 11:57 AM Sayat Satybaldiyev <[hidden email]> wrote:

Thank you for this information. @Yun is there an easy way to expose a number of records in rockdsdb?

On Wed, Sep 26, 2018 at 9:47 AM Yun Tang <[hidden email]> wrote:

Hi Sayat

Before this future is on, you could also find some metrics information, such as hit/miss count, file status from RocksDB itself. By default, RocksDB will dump its stats to its information LOG file every 10 minutes (you could call DBOptions.setStatsDumpPeriodSec to reduce the time interval), and you could find the information LOG file under rocksDB state-backend's db folder.

Best

Yun

---

From: Stefan Richter <[hidden email]>
Sent: Wednesday, September 26, 2018 0:56
To: Sayat Satybaldiyev
Cc: [hidden email]
Subject: Re: Rocksdb Metrics

 

Hi,

this feature is tracked here https://issues.apache.org/jira/browse/FLINK-10423

Best,

Stefan

Am 25.09.2018 um 17:51 schrieb Sayat Satybaldiyev <[hidden email]>:

Flink provides a rich number of metrics. However, I didn't find any metrics for rocksdb state backend not in metrics doc nor in JMX Mbean. 

Is there are any metrics for the rocksdb backend that Flink exposes?