How to submit Flink program to Yarn without upload the fat jar?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to submit Flink program to Yarn without upload the fat jar?

徐涛
Hi Experts,
        When submitting a Flink program to Yarn, the app jar( a fat jar about 200M with Flink dependencies ) will be uploaded to Yarn, which will take a lot of time. I check the code in CliFrontend, and found that there is a config item named “yarn.per-job-cluster.include-user-jar”, I try to set the config item value to “DISABLED”, the fat jar will not be uploaded, but I have to set the CLASSPATH environment of the Yarn container, to point to a route on HDFS. Also I found a config item starts with “containerized.master.env.”, I set the “containerized.master.env.CLASSPATH” value to "hdfs://xm-hdfs-test-01/henry/flink-application-0.0.1-SNAPSHOT-uber.jar”, but with no luck. I also found that the ConfigConstants.CONTAINERIZED_MASTER_ENV_PREFIX variable is marked as deprecated, so maybe I can not use it.
        So how could this requirement be implemented? Because I think it will save a lot of time to start the Flink program.

Best
Henry
Reply | Threaded
Open this post in threaded view
|

Re: How to submit Flink program to Yarn without upload the fat jar?

Chesnay Schepler
Which Flink version are you using? The DISABLED value has not been
working since 1.5, so you may be stuck with uploading the app jar every
time.

On 04/04/2019 11:35, 徐涛 wrote:
> Hi Experts,
> When submitting a Flink program to Yarn, the app jar( a fat jar about 200M with Flink dependencies ) will be uploaded to Yarn, which will take a lot of time. I check the code in CliFrontend, and found that there is a config item named “yarn.per-job-cluster.include-user-jar”, I try to set the config item value to “DISABLED”, the fat jar will not be uploaded, but I have to set the CLASSPATH environment of the Yarn container, to point to a route on HDFS. Also I found a config item starts with “containerized.master.env.”, I set the “containerized.master.env.CLASSPATH” value to "hdfs://xm-hdfs-test-01/henry/flink-application-0.0.1-SNAPSHOT-uber.jar”, but with no luck. I also found that the ConfigConstants.CONTAINERIZED_MASTER_ENV_PREFIX variable is marked as deprecated, so maybe I can not use it.
> So how could this requirement be implemented? Because I think it will save a lot of time to start the Flink program.
>
> Best
> Henry