flink rocksdb where to configure mount point

[Siew Wai Yow]

Hi,

We configure rocksdb as statebackend and checkpoint dir persists to hdfs. When the job is run, rocksdb automatically mount to tmpfs /tmp, which consume memory.

            RocksDBStateBackend rocksdb = new RocksDBStateBackend(new FsStateBackend(hdfs://), true);
            env.setStateBackend(rocksdb);

Questions,

1. What happen if large state is larger than the tmpfs memory size? Will flink do something?
2. Is it possible to configure rocksdb dir via flink?

Thank you.

Regards,

Yow

[Yun Tang]

Hi Siew,

  As far as I know, if the state size is larger than the tmpfs memory size, which means tmpfs will be occupied fully leading to "IOException: No space left on device". However, you just need to call RocksDBStateBackend#setDbStoragePaths method to re-configure the local DB storage dir to a larger disk volume and don't worry about the data completeness as the doc of RocksDBStateBackend#getDbStoragePaths method said:

  "Under these directories on the TaskManager, RocksDB stores its SST files and metadata files. These directories do not need to be persistent, they can be ephermeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints."

Regards,

Yun Tang

---

发件人: Siew Wai Yow <[hidden email]>
发送时间: 2018年7月6日 13:58
收件人: [hidden email]
主题: flink rocksdb where to configure mount point

 

Hi,

We configure rocksdb as statebackend and checkpoint dir persists to hdfs. When the job is run, rocksdb automatically mount to tmpfs /tmp, which consume memory.

            RocksDBStateBackend rocksdb = new RocksDBStateBackend(new FsStateBackend(hdfs://), true);
            env.setStateBackend(rocksdb);

Questions,

1. What happen if large state is larger than the tmpfs memory size? Will flink do something?
2. Is it possible to configure rocksdb dir via flink?

Thank you.

Regards,

Yow

[Siew Wai Yow]

Hi Yun Tang,

Thanks to your reply, this is exactly the answer we need.

also thanks @CongXian's reply.

-Siew Wai

---

From: Tang Cloud <[hidden email]>
Sent: Monday, July 9, 2018 11:05 AM
To: Siew Wai Yow; [hidden email]
Subject: 答复: flink rocksdb where to configure mount point

 

Hi Siew,

  As far as I know, if the state size is larger than the tmpfs memory size, which means tmpfs will be occupied fully leading to "IOException: No space left on device". However, you just need to call RocksDBStateBackend#setDbStoragePaths method to re-configure the local DB storage dir to a larger disk volume and don't worry about the data completeness as the doc of RocksDBStateBackend#getDbStoragePaths method said:

  "Under these directories on the TaskManager, RocksDB stores its SST files and metadata files. These directories do not need to be persistent, they can be ephermeral, meaning that they are lost on a machine failure, because state in RocksDB is persisted in checkpoints."

Regards,

Yun Tang

---

发件人: Siew Wai Yow <[hidden email]>
发送时间: 2018年7月6日 13:58
收件人: [hidden email]
主题: flink rocksdb where to configure mount point

 

Hi,

We configure rocksdb as statebackend and checkpoint dir persists to hdfs. When the job is run, rocksdb automatically mount to tmpfs /tmp, which consume memory.

            RocksDBStateBackend rocksdb = new RocksDBStateBackend(new FsStateBackend(hdfs://), true);
            env.setStateBackend(rocksdb);

Questions,

1. What happen if large state is larger than the tmpfs memory size? Will flink do something?
2. Is it possible to configure rocksdb dir via flink?

Thank you.

Regards,

Yow