Loading... |
Reply to author |
Edit post |
Move post |
Delete this post |
Delete this post and replies |
Change post date |
Print post |
Permalink |
Raw mail |
Hi, I tried to use the newly-supported RocksDB backend (Flink 1.1.4) on my Windows laptop. However, it is not creating any state and is throwing NPE while trying to call: org.apache.flink.streaming.util.HDFSCopyFromLocal$1.run(HDFSCopyFromLocal.java:47) which eventually gets down to not finding Hadoop Winutils. I am not using an hdfs path.... String statepath = parameterTool.get("statepath", "<a href="file:///c:\temp\flink\checkpoints">file:///c:/temp/flink/checkpoints"); env.setStateBackend(new RocksDBStateBackend(statepath)); I notice that in v1.2.0 the async checkpoint mode has been removed (https://issues.apache.org/jira/browse/FLINK-4340)
In the interim is there any hint to using RocksDB on Windows, or should I just revert to File for my playground? Cheers Dave Torok |
Loading... |
Reply to author |
Edit post |
Move post |
Delete this post |
Delete this post and replies |
Change post date |
Print post |
Permalink |
Raw mail |
Hi Dave! In 1.1.x, the default mode of RocksDB uses some Hadoop utilities, regardless of the filesystem used. I think that was a design mistake and we rectified that in the upcoming 1.2.x For 1.1.4, I would use "enableFullyAsyncSnapchots()" on the RocksDB state backend - that mode should also not depend on Hadoop utilities. Best, Stephan On Fri, Dec 23, 2016 at 7:20 PM, Torok, David <[hidden email]> wrote:
... [show rest of quote] |
Loading... |
Reply to author |
Edit post |
Move post |
Delete this post |
Delete this post and replies |
Change post date |
Print post |
Permalink |
Raw mail |
In reply to this post by Torok, David
Thanks for the prompt reply Stephan It is working perfectly now! Best Regards, Dave >Hi Dave! > >In 1.1.x, the default mode of RocksDB uses some Hadoop utilities, regardless of the filesystem used. I think that was a design mistake
and we rectified that in the upcoming 1.2.x > >For 1.1.4, I would use "enableFullyAsyncSnapchots()" on the RocksDB state backend - that mode should also not depend on Hadoop utilities. > >Best, >Stephan |
Free forum by Nabble | Edit this page |