Rocksdb state directory path in EMR

[Sudan S]

Hi,

I have enabled rocksdb(State store) with s3 (external checkpoint) on EMR .

I am using rocksdb as state store with ValueState and checkpoints are stored in s3.

I am able to see checkpoints in s3 and functionality with respect to state store is working fine.

But i am trying to dissect rocksdb folder structure but i am not able to find them on EMR. I am checking /mnt/yarn folder.

Cn u plz help on where flink persists store on EMR when rocksdb is used as state backend?

---

"The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by replying to this e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business."

---

[Dawid Wysakowicz-2]

Hi,

If I understand you correctly, you want to check the local RocksDB files, right? They are stored locally on each TaskManager in a temporary directory.

This can be configured via "state.backend.rocksdb.localdir"[1]. If not specified it will use the globally defined temporary directory set via "io.tmp.dirs"[2], which defaults to : "'LOCAL_DIRS' on Yarn. '_FLINK_TMP_DIR' on Mesos. System.getProperty("java.io.tmpdir") in standalone."

Best,

Dawid

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#state-backend-rocksdb-localdir

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#io-tmp-dirs

On 22/06/2020 15:22, Sudan S wrote:

Hi,

I have enabled rocksdb(State store) with s3 (external checkpoint) on EMR .

I am using rocksdb as state store with ValueState and checkpoints are stored in s3.

I am able to see checkpoints in s3 and functionality with respect to state store is working fine.

But i am trying to dissect rocksdb folder structure but i am not able to find them on EMR. I am checking /mnt/yarn folder.

Cn u plz help on where flink persists store on EMR when rocksdb is used as state backend?

---

"The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by replying to this e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited. All messages sent to and from this e-mail address may be monitored as permitted by applicable law and regulations to ensure compliance with our internal policies and to protect our business."

---