Support priority of the Flink YARN application in Flink 1.9

[tian boxiu]

To: [hidden email]
Title: Support priority of the Flink YARN application in Flink 1.9

Hello everyone,

Many of our batch jobs have changed the execution engine from spark to flink. The flink is deployed on the yarn cluster. In some scenarios, high-priority core tasks need to be submitted on the preemptive queue. Previously, when running in spark, users configured the spark.yarn.priority to control the priority, ranging from 1 to 9.

Currently, flink 1.9 does not support yarn priority submission. The default priority of yarn submission jobs is 1. Based on this, we implemented the submission of the priority range from 1 to 9(from low to high priority). When the user does not configure the parameters, the priority is still 1. After we push the new feature online in June, The system has been tested to run properly.

So I want to create an issue and push the improvement back to community. Hope someone can review it, thanks~

Boxiu

[Xintong Song]

Thanks for bringing this up, Boxiu.

On Wed, Jul 31, 2019 at 12:22 PM tian boxiu <[hidden email]> wrote:

To: [hidden email]
Title: Support priority of the Flink YARN application in Flink 1.9

Hello everyone,

Many of our batch jobs have changed the execution engine from spark to flink. The flink is deployed on the yarn cluster. In some scenarios, high-priority core tasks need to be submitted on the preemptive queue. Previously, when running in spark, users configured the spark.yarn.priority to control the priority, ranging from 1 to 9.

Currently, flink 1.9 does not support yarn priority submission. The default priority of yarn submission jobs is 1. Based on this, we implemented the submission of the priority range from 1 to 9(from low to high priority). When the user does not configure the parameters, the priority is still 1. After we push the new feature online in June, The system has been tested to run properly.

So I want to create an issue and push the improvement back to community. Hope someone can review it, thanks~

Boxiu

[Fabian Hueske-2]

Hi Boxiu,

This sounds like a good feature.

Please have a look at our contribution guidelines [1].

To propose a feature, you should open a Jira issue [2] and start a discussion there.

Please note that the feature freeze for the Flink 1.9 release happened a few weeks ago.

The community is currently working on fixing blockers and testing the release and won't accept new features.

A new feature would go into the next release.

Best, Fabian

[1] https://flink.apache.org/contributing/contribute-code.html

[2] https://issues.apache.org/jira/projects/FLINK/summary

Am Mi., 31. Juli 2019 um 13:24 Uhr schrieb Xintong Song <[hidden email]>:

Thanks for bringing this up, Boxiu.

On Wed, Jul 31, 2019 at 12:22 PM tian boxiu <[hidden email]> wrote:

To: [hidden email]
Title: Support priority of the Flink YARN application in Flink 1.9

Hello everyone,

Many of our batch jobs have changed the execution engine from spark to flink. The flink is deployed on the yarn cluster. In some scenarios, high-priority core tasks need to be submitted on the preemptive queue. Previously, when running in spark, users configured the spark.yarn.priority to control the priority, ranging from 1 to 9.

Currently, flink 1.9 does not support yarn priority submission. The default priority of yarn submission jobs is 1. Based on this, we implemented the submission of the priority range from 1 to 9(from low to high priority). When the user does not configure the parameters, the priority is still 1. After we push the new feature online in June, The system has been tested to run properly.

So I want to create an issue and push the improvement back to community. Hope someone can review it, thanks~

Boxiu

[tian boxiu]

Hello everyone,

I have created a Jira issue in: https://issues.apache.org/jira/browse/FLINK-13548. Thanks for reviewing it.

Thank you~

Boxiu

Fabian Hueske <[hidden email]> 于2019年8月2日周五 下午4:09写道：

Hi Boxiu,

This sounds like a good feature.

Please have a look at our contribution guidelines [1].

To propose a feature, you should open a Jira issue [2] and start a discussion there.

Please note that the feature freeze for the Flink 1.9 release happened a few weeks ago.

The community is currently working on fixing blockers and testing the release and won't accept new features.

A new feature would go into the next release.

Best, Fabian

[1] https://flink.apache.org/contributing/contribute-code.html

[2] https://issues.apache.org/jira/projects/FLINK/summary

Am Mi., 31. Juli 2019 um 13:24 Uhr schrieb Xintong Song <[hidden email]>:

Thanks for bringing this up, Boxiu.

On Wed, Jul 31, 2019 at 12:22 PM tian boxiu <[hidden email]> wrote:

To: [hidden email]
Title: Support priority of the Flink YARN application in Flink 1.9

Hello everyone,

Many of our batch jobs have changed the execution engine from spark to flink. The flink is deployed on the yarn cluster. In some scenarios, high-priority core tasks need to be submitted on the preemptive queue. Previously, when running in spark, users configured the spark.yarn.priority to control the priority, ranging from 1 to 9.

Currently, flink 1.9 does not support yarn priority submission. The default priority of yarn submission jobs is 1. Based on this, we implemented the submission of the priority range from 1 to 9(from low to high priority). When the user does not configure the parameters, the priority is still 1. After we push the new feature online in June, The system has been tested to run properly.

So I want to create an issue and push the improvement back to community. Hope someone can review it, thanks~

Boxiu