Flink Task Allocation on Yarn

[Marvin777]

Hi all,

In the mode of on yarn, a node may contain more than one container,  is there a scheme for assigning tasks to different nodes.

the version is 1.4.2  

Thanks for your assistance.  

[vino yang]

Hi Marvin,

YARN is a resource management and scheduling framework. 

When you run Flink on YARN, Flink will hand over the container's scheduling tasks to YARN. 

This is also the reason why YARN is used. 

If you want to control the start and stop of TM, then I recommend you use standalone mode and set the number of slots per node to 1.

Thanks, vino.

Marvin777 <[hidden email]> 于2018年10月29日周一 上午10:18写道：

Hi all,

In the mode of on yarn, a node may contain more than one container,  is there a scheme for assigning tasks to different nodes.

the version is 1.4.2  

Thanks for your assistance.  

[Marvin777]

Hi vino,

The current generation environment is on yarn. We do not want to increase the operation and maintenance cost of the Standalone mode.

Is there any other way to make better use of the resources of the yarn cluster, try to allocate tasks to containers on different nodes.

Thanks, Marvin.

vino yang <[hidden email]> 于2018年10月29日周一 下午3:04写道：

Hi Marvin,

YARN is a resource management and scheduling framework. 

When you run Flink on YARN, Flink will hand over the container's scheduling tasks to YARN. 

This is also the reason why YARN is used. 

If you want to control the start and stop of TM, then I recommend you use standalone mode and set the number of slots per node to 1.

Thanks, vino.

Marvin777 <[hidden email]> 于2018年10月29日周一 上午10:18写道：

Hi all,

In the mode of on yarn, a node may contain more than one container,  is there a scheme for assigning tasks to different nodes.

the version is 1.4.2  

Thanks for your assistance.  

[vino yang]

Hi Marvin,

Can you tell me why you want to do this?

Thanks, vino.

Marvin777 <[hidden email]> 于2018年10月30日周二 上午10:24写道：

Hi vino,

The current generation environment is on yarn. We do not want to increase the operation and maintenance cost of the Standalone mode.

Is there any other way to make better use of the resources of the yarn cluster, try to allocate tasks to containers on different nodes.

Thanks, Marvin.

vino yang <[hidden email]> 于2018年10月29日周一 下午3:04写道：

Hi Marvin,

YARN is a resource management and scheduling framework. 

When you run Flink on YARN, Flink will hand over the container's scheduling tasks to YARN. 

This is also the reason why YARN is used. 

If you want to control the start and stop of TM, then I recommend you use standalone mode and set the number of slots per node to 1.

Thanks, vino.

Marvin777 <[hidden email]> 于2018年10月29日周一 上午10:18写道：

Hi all,

In the mode of on yarn, a node may contain more than one container,  is there a scheme for assigning tasks to different nodes.

the version is 1.4.2  

Thanks for your assistance.