Cannot set classpath which can be used before job is submitted to yarn.

[bupt_ljy]

Hi,all

   I’m using “bin/flink run -m yarn-cluster” to run my program on yarn. However, it seems that I can’t add my own files into classpath before the the job is submitted to yarn. For example, I have a conf file, which located in my own conf directory, and I need to load file from the conf directory to initialize some classes instances before it’s submitted to yarn.

  I’m grateful if anyone can give me some help.

  Thanks.

[Hequn Cheng]

Hi bupt,

No sure about the answer. 

Do you mean that you can't read the file from local FS? Have you ever tried load the file through a full path? or you choose a wrong classloader.

Best, Hequn

On Thu, Sep 6, 2018 at 11:01 PM bupt_ljy <[hidden email]> wrote:

Hi,all

   I’m using “bin/flink run -m yarn-cluster” to run my program on yarn. However, it seems that I can’t add my own files into classpath before the the job is submitted to yarn. For example, I have a conf file, which located in my own conf directory, and I need to load file from the conf directory to initialize some classes instances before it’s submitted to yarn.

  I’m grateful if anyone can give me some help.

  Thanks.

[bupt_ljy]

Hi Hequn,

I can read it by using the full path, but I want it to be in program's classpath. For example, I use “bin/flink run -m yarn-cluster” to run the program on Server1, and I have a conf file “config.conf" located in “/server/conf” on Server1, I can’t read this file by using ConfigFactory.load(“config.conf”) unless I make changes to the “bin/flink” shell like:

FLOW_CLASSPATH="/server/conf"

exec $JAVA_RUN $JVM_ARGS "${log_setting[@]}" -classpath "`manglePathList "$FLOW_CLASSPATH:$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.client.cli.CliFrontend "$@"

 Original Message 

Sender: Hequn Cheng<[hidden email]>

Recipient: bupt_ljy<[hidden email]>

Cc: user<[hidden email]>

Date: Friday, Sep 7, 2018 09:29

Subject: Re: Cannot set classpath which can be used before job is submitted to yarn.

Hi bupt,

No sure about the answer. 

Do you mean that you can't read the file from local FS? Have you ever tried load the file through a full path? or you choose a wrong classloader.

Best, Hequn

On Thu, Sep 6, 2018 at 11:01 PM bupt_ljy <[hidden email]> wrote:

Hi,all

   I’m using “bin/flink run -m yarn-cluster” to run my program on yarn. However, it seems that I can’t add my own files into classpath before the the job is submitted to yarn. For example, I have a conf file, which located in my own conf directory, and I need to load file from the conf directory to initialize some classes instances before it’s submitted to yarn.

  I’m grateful if anyone can give me some help.

  Thanks.