Odd disk going up and down behavior

Posted by Rex Fenley on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Odd-disk-going-up-and-down-behavior-tp40519.html

We have a job with a bunch of joins and maps from Kafka to Elasticsearch. At some point later into the initial indexing process we began seeing these odd spikey graphs below. We're curious what could have started causing this?

One interesting facet is disk usage drops an order of magnitude (.5 - 2 GiB) while running and then fires back up, holds steady during checkpointing, and then drops again during normal running (we're also trying to reduce our checkpointing times which are ~1-3 min, but that's a separate topic).

Is RocksDB compaction mechanism at play when the disk usage drops? You can see this behavior in the first 2 graphs.

Other graphs

--