Unable to deduce RocksDB api calls in streaming.

[Aviral Srivastava]

Hi all!

I want to profile the time taken to make snapshot calls to RocksDB when using Flink in streaming mode.

I have forked the flink core repo, added the example of fraud detection, configured the state backend and checkpointing. The program is running successfully.

I have also been able to import the code base into IntelliJ IDE and debug it. However, I could not trace the function calls to the RocksDB. Is there any documentation that would help me out in this regard? Or can someone help me here?

Corresponding question on SO: https://stackoverflow.com/questions/63103295/what-are-the-api-calls-made-to-create-snapshot-while-using-rocksdb-as-the-state

[Timo Walther]

Hi Aviral,

as far as I know we are not calling RocksDB API to perform snapshots. As
the Stackoverflow answer also indicates most of the snapshotting is done
outside of RocksDB by just dealing with the SST files. Have you checked
the available metrics in the web UI?

https://ci.apache.org/projects/flink/flink-docs-master/monitoring/checkpoint_monitoring.html

But maybe a RocksDB expert (in CC) knows more about this topic.

Regards,
Timo

On 27.07.20 00:02, Aviral Srivastava wrote:

> Hi all!
>
> I want to profile the time taken to make snapshot calls to RocksDB when
> using Flink in streaming mode.
>
> I have forked the flink core repo, added the example of fraud detection,
> configured the state backend and checkpointing. The program is running
> successfully.
>
> I have also been able to import the code base into IntelliJ IDE and
> debug it. However, I could not trace the function calls to the RocksDB.
> Is there any documentation that would help me out in this regard? Or can
> someone help me here?
>
> Corresponding question on SO:
> https://stackoverflow.com/questions/63103295/what-are-the-api-calls-made-to-create-snapshot-while-using-rocksdb-as-the-state
>
> Best,
> Aviral Srivastava
> LinkedIn <https://www.linkedin.com/in/sriavi/> | Website
> <https://aviralsrivastava.com/>