yarn kill container due to running beyond physical memory limits [ How can i debug memory issue ]

Posted by prateekarora on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/yarn-kill-container-due-to-running-beyond-physical-memory-limits-How-can-i-debug-memory-issue-tp7296.html

Hi

I am running flink 1.0.2 with Yarn .

After running application for some time , Yarn kill my container due to running beyond physical memory limits .

how can i debug memory issue ?

below are the logs :

Container container_1463184272818_0165_01_000012 is completed with diagnostics: Container [pid=19349,containerID=container_1463184272818_0165_01_000012] is running beyond physical memory limits. Current usage: 6.0 GB of 6 GB physical memory used; 9.1 GB of 12.6 GB virtual memory used. Killing container.

Dump of the process-tree for container_1463184272818_0165_01_000012 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 19356 19349 19349 19349 (java) 39350 9110 9711140864 1581168 /usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m -XX:MaxDirectMemorySize=4608m -Djava.library.path=/home/nativelibraries/native_lib/ -Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
        |- 19349 19345 19349 19349 (bash) 0 0 11456512 359 /bin/bash -c /usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m -XX:MaxDirectMemorySize=4608m -Djava.library.path=/home/nativelibraries/native_lib/ -Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1> /var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.out 2> /var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.err

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143