RocksDB Windows / Flink 1.1.4 - requires Hadoop?

[Torok, David]

Hi,

 

I tried to use the newly-supported RocksDB backend (Flink 1.1.4) on my Windows laptop.

 

However, it is not creating any state and is throwing NPE while trying to call:

 

org.apache.flink.streaming.util.HDFSCopyFromLocal$1.run(HDFSCopyFromLocal.java:47)

 

which eventually gets down to not finding Hadoop Winutils.

 

I am not using an hdfs path.... 

 

String statepath = parameterTool.get("statepath", "<a href="file:///c:\temp\flink\checkpoints">file:///c:/temp/flink/checkpoints");

env.setStateBackend(new RocksDBStateBackend(statepath));

                               

I notice that in v1.2.0 the async checkpoint mode has been removed

(https://issues.apache.org/jira/browse/FLINK-4340)

 

In the interim is there any hint to using RocksDB on Windows, or should I just revert to File for my playground?

 

Cheers

Dave Torok

 

[Stephan Ewen]

Hi Dave!

In 1.1.x, the default mode of RocksDB uses some Hadoop utilities, regardless of the filesystem used. I think that was a design mistake and we rectified that in the upcoming 1.2.x

For 1.1.4, I would use "enableFullyAsyncSnapchots()" on the RocksDB state backend - that mode should also not depend on Hadoop utilities.

Best,

Stephan

On Fri, Dec 23, 2016 at 7:20 PM, Torok, David <[hidden email]> wrote:

Hi,

 

I tried to use the newly-supported RocksDB backend (Flink 1.1.4) on my Windows laptop.

 

However, it is not creating any state and is throwing NPE while trying to call:

 

org.apache.flink.streaming.util.HDFSCopyFromLocal$1.run(HDFSCopyFromLocal.java:47)

 

which eventually gets down to not finding Hadoop Winutils.

 

I am not using an hdfs path.... 

 

String statepath = parameterTool.get("statepath", "file:///c:/temp/flink/checkpoints");

env.setStateBackend(new RocksDBStateBackend(statepath));

                               

I notice that in v1.2.0 the async checkpoint mode has been removed

(https://issues.apache.org/jira/browse/FLINK-4340)

 

In the interim is there any hint to using RocksDB on Windows, or should I just revert to File for my playground?

 

Cheers

Dave Torok

 

[Torok, David]

Thanks for the prompt reply Stephan

 

It is working perfectly now!

 

Best Regards,

Dave

 

 

>Hi Dave!

> 

>In 1.1.x, the default mode of RocksDB uses some Hadoop utilities, regardless of the filesystem used. I think that was a design mistake and we rectified that in the upcoming 1.2.x

> 

>For 1.1.4, I would use "enableFullyAsyncSnapchots()" on the RocksDB state backend - that mode should also not depend on Hadoop utilities.

> 

>Best,

>Stephan