Hi,
You have to enable CPU scheduling in YARN, otherwise it always
shows that only 1 CPU is allocated for each container,
regardless of how many Flink try to allocated.
TaskManager memory is 1400MB, but Flink reserves some amount for
for off-heap memory, so the actual heap size is smaller.
This is controlled by 2 settings:
containerized.heap-cutoff-min:
default 600MB
containerized.heap-cutoff-ratio:
default 15% of TM's memory
That's why your TM's heap size is limitted to ~800MB (1400 - 600)
Regards,
Kien
On 4/29/2018 12:36 PM, Soheil
Pourbafrani wrote:
Hi, I have an Flink .jar file and I submit it on
yarn cluster using the command:
flink run -m yarn-cluster -yn 5 -yjm 768 -ytm 1400 -ys 2
-yqu streamQ my_program.jar
According to the submitting command I expect:
// It will create 5 containers ----> satisfied
// each container should use 2 core --> not satisfied
according to the YARN webui but in jobmanager webui it shows
each taskmanager has 2 slot
// jobmanager should use 768m ram --> satisfied
// each taskmanager should use 1400m ram --> satisfied
according to the YARN webui but in jobmanager webui it
shows for each taskmanager: 767m heap memory and 478m used by
Flink
Why the number of used cores and memory is not as desired?
Locked
