Fw: 1.4.2 - Unable to start YARN containers with more than 1 vCores per Task Manager

[Ashish Pokharel]

Hi All,

 

I had been using the following command in a Lab environment successfully in 1.3 Flink version.

 

yarn-session.sh -n 4 -s 4 -jm 2048 -tm 2048 -Dyarn.containers.vcores=2 -nm infra.test3

 

As expected, I see 4 TMs with 16 slots and taking 8 vCores from YARN. In a new Prod environment, I am using 1.4.2 version and I am only seeing 1 vCore per container with the same command. I did check YARN config and max vCores allowed per container is configured as 48.

 

Anyone has seen something similar or has idea on what we can look into? Might be YARN config?

 

Thanks,

 

Ashish

[shashank734]

CHeck in your Yarn configuration, Are you using DeafaultResourceCalculater
which only uses memory while allocating resources. So you have to change it
to DominantResourceCalculator.

<property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
   
<value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
</property>



Check this:

https://hortonworks.com/blog/managing-cpu-resources-in-your-hadoop-yarn-clusters/



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Ashish Pokharel]

Thanks Shashank, I will check ot out.

-- Ashish

On Tue, Apr 3, 2018 at 10:11 AM, shashank734

<[hidden email]> wrote:

CHeck in your Yarn configuration, Are you using DeafaultResourceCalculater
which only uses memory while allocating resources. So you have to change it
to DominantResourceCalculator.

<property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
 
<value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
</property>



Check this:

https://hortonworks.com/blog/managing-cpu-resources-in-your-hadoop-yarn-clusters/



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/