Each yarn container only use 1 vcore even if taskmanager.numberOfTaskSlots is set

[徐涛]

Hi Experts,
        I am running Flink 1.7.1 program on Yarn 2.7, the taskmanager.numberOfTaskSlots is set to 4. The parallelism.default is set to 8. When the program is running, 3 yarn containers is launched, but each of them only use 1 vcore, I think by default the number of vcores is set to the number of slots per TaskManager.
        Because it can not get enough CPU resources, the CPU load may be high. So how can this problem be fixed?
        Thanks a lot for your help.

Best
Henry

[Gary Yao-4]

Hi Henry,

If I understand you correctly, you want YARN to allocate 4 vcores per TM
container. You can achieve this by enabling the FairScheduler in YARN [1][2].

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#yarn-containers-vcores
[2] https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

On Mon, Feb 18, 2019 at 3:49 AM 徐涛 <[hidden email]> wrote:

Hi Experts,
        I am running Flink 1.7.1 program on Yarn 2.7, the taskmanager.numberOfTaskSlots is set to 4. The parallelism.default is set to 8. When the program is running, 3 yarn containers is launched, but each of them only use 1 vcore, I think by default the number of vcores is set to the number of slots per TaskManager.
        Because it can not get enough CPU resources, the CPU load may be high. So how can this problem be fixed?
        Thanks a lot for your help.

Best
Henry