EMR vCores and slot allocation

[Ken Krugler]

Based on what Flink reports in the JobManager GUI, it looks like it thinks that the EC2 instances I’m using for my EMR jobs only have 4 physical cores.

Which would make sense, as Amazon describes these servers as having 8 vCores.

From https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html, the recommended configuration would then be 4 slots/TaskManager, yes?

Thanks,

— Ken

--------------------------

Ken Krugler

[phone]

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[rmetzger0]

Hi Ken,

sorry for the late response. The number of CPU cores we show in the web interface is based on what the JVM tells us from "Runtime.getRuntime().availableProcessors();". I'm not sure how tthe processor count behaves on Amazon VMs.

Given that each of your servers has 8 vCores, I would set the slot count to 8 per Taskmanager.

On Fri, Apr 29, 2016 at 1:58 AM, Ken Krugler <[hidden email]> wrote:

Based on what Flink reports in the JobManager GUI, it looks like it thinks that the EC2 instances I’m using for my EMR jobs only have 4 physical cores.

Which would make sense, as Amazon describes these servers as having 8 vCores.

From https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html, the recommended configuration would then be 4 slots/TaskManager, yes?

Thanks,

— Ken

--------------------------

Ken Krugler

<a href="tel:%2B1%20530-210-6378" value="[phone]" target="_blank">[phone]

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[Fabian Hueske-2]

The slot configuration should depend on the complexity of jobs.

Since each slot runs a "slice" of a program, one slot might potentially execute many concurrent tasks.

For complex jobs you should allocate more than one core for each slot.

2016-05-02 10:12 GMT+02:00 Robert Metzger <[hidden email]>:

Hi Ken,

sorry for the late response. The number of CPU cores we show in the web interface is based on what the JVM tells us from "Runtime.getRuntime().availableProcessors();". I'm not sure how tthe processor count behaves on Amazon VMs.

Given that each of your servers has 8 vCores, I would set the slot count to 8 per Taskmanager.

On Fri, Apr 29, 2016 at 1:58 AM, Ken Krugler <[hidden email]> wrote:

Based on what Flink reports in the JobManager GUI, it looks like it thinks that the EC2 instances I’m using for my EMR jobs only have 4 physical cores.

Which would make sense, as Amazon describes these servers as having 8 vCores.

From https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html, the recommended configuration would then be 4 slots/TaskManager, yes?

Thanks,

— Ken

--------------------------

Ken Krugler

<a href="tel:%2B1%20530-210-6378" value="[phone]" target="_blank">[phone]

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr