confusing RocksDBStateBackend parameters

[bowen.li]

Hello guys,

      I've been trying to figure out differences among several parameters of RocksDBStateBackend. The confusing parameters are:

      In flink-conf.yaml:

      1. state.backend.fs.checkpointdir

      2. state.backend.rocksdb.checkpointdir

      3. state.checkpoints.dir

      and 

       4. the param 'checkpointDataUri' you pass in to RocksDBStateBackend

constructor in`public RocksDBStateBackend(URI checkpointDataUri)`

        This email thread explained the first three well. But what's the 4th one for? What's its difference from others? I'd appreciate your clarification.

       Thanks very much!

Bowen

        

[mmziyad]

Hi,

To set the rocksdb state, you have two options:

1. Set the default state of the flink cluster, using the below parameters in flink-conf.yaml file

state.backend: rocksdb

state.backend.fs.checkpointdir: hdfs://namenode:40010/flink/checkpoints

2. Set a per job state backend (which overrides the default setting)

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

env.setStateBackend(new RocksDBStateBackend(parameterTool.getRequired("checkpointDataUri")));

Here you specify the path (for example, an HDFS directory like hdfs://namenode:40010/flink/checkpoints) as the checkpointDataURI.

Hope that helps.

Best

Ziyad

On Fri, Jun 16, 2017 at 8:20 PM, Bowen Li <[hidden email]> wrote:

Hello guys,

      I've been trying to figure out differences among several parameters of RocksDBStateBackend. The confusing parameters are:

      In flink-conf.yaml:

      1. state.backend.fs.checkpointdir

      2. state.backend.rocksdb.checkpointdir

      3. state.checkpoints.dir

      and 

       4. the param 'checkpointDataUri' you pass in to RocksDBStateBackend

constructor in`public RocksDBStateBackend(URI checkpointDataUri)`

        This email thread explained the first three well. But what's the 4th one for? What's its difference from others? I'd appreciate your clarification.

       Thanks very much!

Bowen

        

[bowen.li]

Thanks for your clarification, Ziyad! I will try it out.

On Sat, Jun 17, 2017 at 3:45 PM, Ziyad Muhammed <[hidden email]> wrote:

Hi,

To set the rocksdb state, you have two options:

1. Set the default state of the flink cluster, using the below parameters in flink-conf.yaml file

state.backend: rocksdb

state.backend.fs.checkpointdir: hdfs://namenode:40010/flink/checkpoints

2. Set a per job state backend (which overrides the default setting)

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

env.setStateBackend(new RocksDBStateBackend(parameterTool.getRequired("checkpointDataUri")));

Here you specify the path (for example, an HDFS directory like hdfs://namenode:40010/flink/checkpoints) as the checkpointDataURI.

Hope that helps.

Best

Ziyad

On Fri, Jun 16, 2017 at 8:20 PM, Bowen Li <[hidden email]> wrote:

Hello guys,

      I've been trying to figure out differences among several parameters of RocksDBStateBackend. The confusing parameters are:

      In flink-conf.yaml:

      1. state.backend.fs.checkpointdir

      2. state.backend.rocksdb.checkpointdir

      3. state.checkpoints.dir

      and 

       4. the param 'checkpointDataUri' you pass in to RocksDBStateBackend

constructor in`public RocksDBStateBackend(URI checkpointDataUri)`

        This email thread explained the first three well. But what's the 4th one for? What's its difference from others? I'd appreciate your clarification.

       Thanks very much!

Bowen