Re: adding core-site xml to flink1.11

Posted by rmetzger0 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/adding-core-site-xml-to-flink1-11-tp38933p38981.html

Hi Shachar,

Why do you want to use the core-site.xml to configure the file system?

Since we are adding the file systems as plugins, their initialization is customized. It might be the case that we are intentionally ignoring xml configurations from the classpath.

You can configure the filesystem in the flink-conf.yaml file.

On Sun, Oct 25, 2020 at 7:56 AM Shachar Carmeli <[hidden email]> wrote:

Hi,
I'm trying to define filesystem to flink 1.11 using core-site.xml
I tried adding in the flink-conf.yaml env.hadoop.conf.dir and I see it is added to the classpath
also adding environment variable HADOOP_CONF_DIR didn't help

The flink 1.11.2 is running on docker using kubernetes

I added hadoop using plugin as mentioned in https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins

when configure the parameters manually I can connect to the local s3a server
So it looks like the flink is not reading the core-site.xml file

please advise

Thanks,
Shachar