Flink on k8s

[art]

Hi

    How to configure statebackend when I deploy flink on k8s , I just add the following to flink-conf.yaml, but it doesn’t work

state.backend: rocksdb

state.checkpoints.dir: hdfs://slave2:8020/flink/checkpoints

state.savepoints.dir: hdfs://slave2:8020/flink/savepoints

state.backend.incremental: true

	superainbower
superainbower@...

签名由 网易邮箱大师 定制

[art]

And I got this error log 

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please seehttps://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.

	superainbower
superainbower@...

签名由 网易邮箱大师 定制

On 09/30/2020 14:48，[hidden email] wrote：

Hi

    How to configure statebackend when I deploy flink on k8s , I just add the following to flink-conf.yaml, but it doesn’t work

state.backend: rocksdb

state.checkpoints.dir: hdfs://slave2:8020/flink/checkpoints

state.savepoints.dir: hdfs://slave2:8020/flink/savepoints

state.backend.incremental: true

	superainbower
superainbower@...

签名由 网易邮箱大师 定制

[Arvid Heise-3]

Hi,

you are missing the Hadoop libraries, hence there is no hdfs support.

In Flink 1.10 and earlier, you would simply copy flink-shaded-hadoop-2-uber[1] into your opt/ folder. However, since Flink 1.11, we recommend to install Hadoop and point to it with HADOOP_CLASSPATH.

Now, the latter approach does not seem to be terribly meaningful to K8s. You can still use the old flink-shaded-hadoop-2-uber for now but it's better to have a different way in the future.

[hidden email] , do you know what's the current recommended approach? Should we release a light-weight hdfs filesystem to make it symmetric to the other filesystems?

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html#using-flink-shaded-hadoop-2-uber-jar-for-resolving-dependency-conflicts-legacy

On Wed, Sep 30, 2020 at 9:02 AM superainbower <[hidden email]> wrote:

And I got this error log 

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please seehttps://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.

	superainbower
superainbower@...

签名由 网易邮箱大师 定制

On 09/30/2020 14:48，[hidden email] wrote：

Hi

    How to configure statebackend when I deploy flink on k8s , I just add the following to flink-conf.yaml, but it doesn’t work

state.backend: rocksdb

state.checkpoints.dir: hdfs://slave2:8020/flink/checkpoints

state.savepoints.dir: hdfs://slave2:8020/flink/savepoints

state.backend.incremental: true

	superainbower
superainbower@...

签名由 网易邮箱大师 定制

--