Hi Zheng,
To divide managed memory for operators[1], we need to consider which tasks will
run in the same slot. In batch jobs, vertices in different regions may not run at
the same time. If we put them in the same slot sharing group, running tasks
may run slower with less managed memory, while managed memory reserved
for tasks that are not running yet will be wasted.
More slots can be requested as a result. However, it's not necessary to add more
containers. One slot will serve fewer tasks, this means you can decrease the slot
size (via increase "taskmanager.numberOfTaskSlots") so that the previous number
of containers can be enough. This is because you are still running that many tasks
at the same time, although they are spread into more slots.
Hi All,
I find we set AllVerticesInSameSlotSharingGroupByDefault to false in flink 1.10. It will make batch job request lots of containers. I'm not sure why we set it to false directly. I try to set it to true and find the batch job can
run correctly with a small amount containers. Why don't we add a configuration to let user configure it?
Best,
Faaron Zheng
Locked
