Flink configuration

[Nancy Estrada]

Hi all,

I have been reading about how to configure Flink when we have a set up consisting on a couple of VMs with more than 1 vCore.  I am a bit confused about how to set the degree of parallelism in the taskmanager.numberOfTaskSlots parameter:

* According to the Flink documentation[1], this value is typically proportional to the number of physical CPU cores that the TaskManager’s machine has.

* However the YARN documentation[2], makes reference to the number of Virtual CPU cores per TaskManager.

My question is, If my Flink Jobs will be running on VMs (without using YARN),  the "taskmanager.numberOfTaskSlots" will depend on the number of vCPU that mi VM has? or must be related to the physical cores?  

Thanks in advance for your help!
Nancy

[1]https://ci.apache.org/projects/flink/flink-docs-release-0.8/config.html
[2]https://ci.apache.org/projects/flink/flink-docs-release-0.8/yarn_setup.html

[Aljoscha Krettek]

Hi,

that wording is from a time where no-one though about VMs with virtual cores. IMHO this maps directly to virtual cores so you should set it according to the number of virtual cores of your VMs.

Cheers,

Aljoscha

On Mon, 23 Jan 2017 at 11:51 Nancy Estrada <[hidden email]> wrote:

Hi all,

I have been reading about how to configure Flink when we have a set up
consisting on a couple of VMs with more than 1 vCore.  I am a bit confused
about how to set the degree of parallelism in the
taskmanager.numberOfTaskSlots parameter:

* According to the Flink documentation[1], this value is typically
proportional to the number of/ physical CPU cores/ that the TaskManager’s
machine has.

* However the YARN documentation[2], makes reference to the number of
/Virtual CPU cores/ per TaskManager.

My question is, If my Flink Jobs will be running on VMs (without using
YARN),  the "taskmanager.numberOfTaskSlots" will depend on the number of
vCPU that mi VM has? or must be related to the physical cores?

Thanks in advance for your help!
Nancy

[1]https://ci.apache.org/projects/flink/flink-docs-release-0.8/config.html
[2]https://ci.apache.org/projects/flink/flink-docs-release-0.8/yarn_setup.html



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-configuration-tp11210.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Greg Hogan]

Has anyone reported decreased performance with hyper-threading?

On Tue, Jan 24, 2017 at 11:18 AM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

that wording is from a time where no-one though about VMs with virtual cores. IMHO this maps directly to virtual cores so you should set it according to the number of virtual cores of your VMs.

Cheers,

Aljoscha

On Mon, 23 Jan 2017 at 11:51 Nancy Estrada <[hidden email]> wrote:

Hi all,

I have been reading about how to configure Flink when we have a set up
consisting on a couple of VMs with more than 1 vCore.  I am a bit confused
about how to set the degree of parallelism in the
taskmanager.numberOfTaskSlots parameter:

* According to the Flink documentation[1], this value is typically
proportional to the number of/ physical CPU cores/ that the TaskManager’s
machine has.

* However the YARN documentation[2], makes reference to the number of
/Virtual CPU cores/ per TaskManager.

My question is, If my Flink Jobs will be running on VMs (without using
YARN),  the "taskmanager.numberOfTaskSlots" will depend on the number of
vCPU that mi VM has? or must be related to the physical cores?

Thanks in advance for your help!
Nancy

[1]https://ci.apache.org/projects/flink/flink-docs-release-0.8/config.html
[2]https://ci.apache.org/projects/flink/flink-docs-release-0.8/yarn_setup.html



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-configuration-tp11210.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.