Re: Substasks - Uneven allocation

Posted by Kien Truong on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Substasks-Uneven-allocation-tp19612p19642.html

Hi Pedro,

You can try to call either

.rebalance() or .shuffle()

before the Async operator.


Shuffle might give a better result if you have fewer tasks than parallelism.

Best regards,
Kien

On 4/18/2018 11:10 PM, PedroMrChaves wrote:

Hello,

I have a job that has one async operational node (i.e. implements
AsyncFunction). This Operational node will spawn multiple threads that
perform heavy tasks (cpu bound). 

I have a Flink Standalone cluster deployed on two machines of 32 cores and
128 gb of RAM, each machine has one task manager and one Job Manager. When I
deploy the job, all of the subtasks from the async operational node end up
on the same machine, which causes it to have a much higher cpu load then the
other. 

I've researched ways to overcome this issue, but I haven't found a solution
to my problem. 
Ideally, the subtasks would be evenly split across both machines. 

Can this problem be solved somehow? 

Regards,
Pedro Chaves. 



-----
Best Regards,
Pedro Chaves
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/