RocksDB checkpointing dir per TM

[Taher Koitawala]

Hi All,

          Our current cluster configuration uses one HDD which is mainly for root and an other NVME disk per node, [1]we want make sure all TMs write their own RocksDB files to the NVME disk only, how do we do that? 

[2] Is it also possible to specify multiple directories per TMs so that we have an even spread when the RocksDB files are written?  

Thanks,

Taher Koitawala 

[Andrey Zagrebin]

Hi Taher,

TMs keep state locally while running, in this case RocksDB files already belong to TM.
You can point it to the same NVME disk location on each node, relevant Flink options here are:
- io.tmp.dirs
- taskmanager.state.local.root-dirs
This data is transient and has temporary nature. It does not survive a job failure.

The checkpoint is a logical snapshot of the operator state for all involved TMs,
so it belongs to the job and usually uploaded to a distributed file system available on all TMs.
The location is set in Flink option ‘state.checkpoints.dir'.
This way job can restore from it with different set of TMs.

Best,
Andrey

> On 26 Oct 2018, at 08:29, Taher Koitawala <[hidden email]> wrote:
>
> Hi All,
>           Our current cluster configuration uses one HDD which is mainly for root and an other NVME disk per node, [1]we want make sure all TMs write their own RocksDB files to the NVME disk only, how do we do that?
>
> [2] Is it also possible to specify multiple directories per TMs so that we have an even spread when the RocksDB files are written?  
>
> Thanks,
> Taher Koitawala

[Taher Koitawala]

Thanks! 

On Fri 26 Oct, 2018, 3:31 PM Andrey Zagrebin, <[hidden email]> wrote:

Hi Taher,

TMs keep state locally while running, in this case RocksDB files already belong to TM.
You can point it to the same NVME disk location on each node, relevant Flink options here are:
- io.tmp.dirs
- taskmanager.state.local.root-dirs
This data is transient and has temporary nature. It does not survive a job failure.

The checkpoint is a logical snapshot of the operator state for all involved TMs,
so it belongs to the job and usually uploaded to a distributed file system available on all TMs.
The location is set in Flink option ‘state.checkpoints.dir'.
This way job can restore from it with different set of TMs.

Best,
Andrey

> On 26 Oct 2018, at 08:29, Taher Koitawala <[hidden email]> wrote:
>
> Hi All,
>           Our current cluster configuration uses one HDD which is mainly for root and an other NVME disk per node, [1]we want make sure all TMs write their own RocksDB files to the NVME disk only, how do we do that?
>
> [2] Is it also possible to specify multiple directories per TMs so that we have an even spread when the RocksDB files are written? 
>
> Thanks,
> Taher Koitawala

[Elias Levy]

There is also state.backend.rocksdb.localdir.  Oddly, I can find the documentation for it in the 1.5 docs, but not in the 1.6 docs.  The option is still in master, and it is used.

On Fri, Oct 26, 2018 at 3:01 AM Andrey Zagrebin <[hidden email]> wrote:

Hi Taher,

TMs keep state locally while running, in this case RocksDB files already belong to TM.
You can point it to the same NVME disk location on each node, relevant Flink options here are:
- io.tmp.dirs
- taskmanager.state.local.root-dirs
This data is transient and has temporary nature. It does not survive a job failure.

The checkpoint is a logical snapshot of the operator state for all involved TMs,
so it belongs to the job and usually uploaded to a distributed file system available on all TMs.
The location is set in Flink option ‘state.checkpoints.dir'.
This way job can restore from it with different set of TMs.

Best,
Andrey

> On 26 Oct 2018, at 08:29, Taher Koitawala <[hidden email]> wrote:
>
> Hi All,
>           Our current cluster configuration uses one HDD which is mainly for root and an other NVME disk per node, [1]we want make sure all TMs write their own RocksDB files to the NVME disk only, how do we do that?
>
> [2] Is it also possible to specify multiple directories per TMs so that we have an even spread when the RocksDB files are written? 
>
> Thanks,
> Taher Koitawala

[Till Rohrmann]

This is a very good point Elias. We actually forgot to add these options to the configuration documentation after a refactoring. I will fix it.

Cheers,

Till

On Fri, Oct 26, 2018 at 8:27 PM Elias Levy <[hidden email]> wrote:

There is also state.backend.rocksdb.localdir.  Oddly, I can find the documentation for it in the 1.5 docs, but not in the 1.6 docs.  The option is still in master, and it is used.

On Fri, Oct 26, 2018 at 3:01 AM Andrey Zagrebin <[hidden email]> wrote:

Hi Taher,

TMs keep state locally while running, in this case RocksDB files already belong to TM.
You can point it to the same NVME disk location on each node, relevant Flink options here are:
- io.tmp.dirs
- taskmanager.state.local.root-dirs
This data is transient and has temporary nature. It does not survive a job failure.

The checkpoint is a logical snapshot of the operator state for all involved TMs,
so it belongs to the job and usually uploaded to a distributed file system available on all TMs.
The location is set in Flink option ‘state.checkpoints.dir'.
This way job can restore from it with different set of TMs.

Best,
Andrey

> On 26 Oct 2018, at 08:29, Taher Koitawala <[hidden email]> wrote:
>
> Hi All,
>           Our current cluster configuration uses one HDD which is mainly for root and an other NVME disk per node, [1]we want make sure all TMs write their own RocksDB files to the NVME disk only, how do we do that?
>
> [2] Is it also possible to specify multiple directories per TMs so that we have an even spread when the RocksDB files are written? 
>
> Thanks,
> Taher Koitawala