Alternatives to FLINK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Alternatives to FLINK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR

Foster, Craig

Can I set these in the configuration file? This would be ideal vs. environment variables for me but I’m not seeing it in the documentation.

 

Thanks,

Craig

 

Reply | Threaded
Open this post in threaded view
|

Re: Alternatives to FLINK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR

Bajaj, Abhinav

I think the FLINK_CONF_DIR points to the conf directory. This is the place where the Flink CLI looks for the flink-conf.yaml file.

 

I think there is an alternate option for HADOOP_CONF_DIR, YARN_CONF_DIR but I am not sure.

Check this https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/config.html#common-options

Config option - “fs.hdfs.hadoopconf:“

 

~ Abhi

 

 

From: "Foster, Craig" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 23, 2017 at 9:43 AM
To: "[hidden email]" <[hidden email]>
Subject: Alternatives to FLINK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR

 

Can I set these in the configuration file? This would be ideal vs. environment variables for me but I’m not seeing it in the documentation.

 

Thanks,

Craig

 

Reply | Threaded
Open this post in threaded view
|

Re: Alternatives to FLINK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR

Foster, Craig

Thanks! I looked in the config.sh file that sort of works with both the configuration file and these environment variables. After inspection, it doesn’t make sense to set FLINK_CONF_DIR in that config file since that location determines where we would look for that file.

 

However, I thought that I could add a patch to this shell script to at least allow someone to set the Hadoop and Yarn configuration directories in the config file.

 

From: "Bajaj, Abhinav" <[hidden email]>
Date: Thursday, March 23, 2017 at 10:42 AM
To: "Foster, Craig" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Alternatives to FLINK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR

 

I think the FLINK_CONF_DIR points to the conf directory. This is the place where the Flink CLI looks for the flink-conf.yaml file.

 

I think there is an alternate option for HADOOP_CONF_DIR, YARN_CONF_DIR but I am not sure.

Check this https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/config.html#common-options

Config option - “fs.hdfs.hadoopconf:“

 

~ Abhi

 

 

From: "Foster, Craig" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, March 23, 2017 at 9:43 AM
To: "[hidden email]" <[hidden email]>
Subject: Alternatives to FLINK_CONF_DIR, HADOOP_CONF_DIR, YARN_CONF_DIR

 

Can I set these in the configuration file? This would be ideal vs. environment variables for me but I’m not seeing it in the documentation.

 

Thanks,

Craig