Balancing operation across machines

[flint-stone]

Hello!

Just wondering if there is a way to load-balancing all executors on one operator across the machine. I have a reduce operator that has less parallelism level than the total number of slots in the cluster. Currently flink tries to compact executors in an operator and schedule them to some of the machines in the cluster. Just wondering if there is a way to make Flink schedule these executor evenly across the machines (e.g. one executor per machine.)?

Thanks!

Le

[Tony Wei]

Hi,

I think this patch[1] is what you need and it might be merged in 1.4.0 release.

AFAIK, there is no way to change the behavior of scheduler from public API. One work around that I could come up with is to give each TM one available slot, make all operators in the same SlotSharingGroup and set the parallelism as the number of TMs.

Hope this will help you.

Best,

Tony Wei

[1] https://issues.apache.org/jira/browse/FLINK-7851

2017-10-22 10:35 GMT+08:00 Le Xu <[hidden email]>:

Hello!

Just wondering if there is a way to load-balancing all executors on one operator across the machine. I have a reduce operator that has less parallelism level than the total number of slots in the cluster. Currently flink tries to compact executors in an operator and schedule them to some of the machines in the cluster. Just wondering if there is a way to make Flink schedule these executor evenly across the machines (e.g. one executor per machine.)?

Thanks!

Le