(DEPRECATED) Apache Flink User Mailing List archive.

Running into memory issues while running on Yarn

classic Classic list List threaded Threaded
♦
♦
 2 messages Options
Sachin Goel
Reply | Threaded
Open this post in threaded view
♦
♦
|

Running into memory issues while running on Yarn

Sachin Goel
Hey!

I'm running locally under this configuration(copied from nodemanager logs):
physical-memory=8192 virtual-memory=17204 virtual-cores=8

Before starting a flink deployment, memory usage stats show 3.7 GB used on system, indicating lots of free memory for flink containers.
However, after I submit using minimal resource requirements, ./yarn-session.sh -n 1 -tm 768, the cluster deploys successfully but then every application on system receives a sigterm and it basically kills the current user session, logging out of the system.

The job manager and task manager logs contain just the information that a SIGTERM was received and shut down gracefully.
All yarn and dfs process contain the log information showing the receipt of a sigterm.

Here's the relevant log from nodemanager:

2017-01-05 17:00:06,089 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1483603191971_0002_01_000002 transitioned from LOCALIZED to RUNNING
2017-01-05 17:00:06,092 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /opt/hadoop-2.7.3/tmp/nm-local-dir/usercache/kirk/appcache/application_1483603191971_0002/container_1483603191971_0002_01_000002/default_container_executor.sh]
2017-01-05 17:00:08,731 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1483603191971_0002_01_000002
2017-01-05 17:00:08,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 17872 for container-id container_1483603191971_0002_01_000001: 282.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used
2017-01-05 17:00:08,744 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_1483603191971_0002_01_000001 has processes older than 1 iteration running over the configured limit. Limit=2254857728, current usage = 2255896576
2017-01-05 17:00:08,745 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=17872,containerID=container_1483603191971_0002_01_000001] is running beyond virtual memory limits. Current usage: 282.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1483603191971_0002_01_000001 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 17872 17870 17872 17872 (bash) 0 0 21409792 812 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Xmx424M  -Dlog.file=/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnApplicationMasterRunner  1>/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.out 2>/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.err 
	|- 17879 17872 17872 17872 (java) 748 20 2234486784 71553 /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Xmx424M -Dlog.file=/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnApplicationMasterRunner 

2017-01-05 17:00:08,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Removed ProcessTree with root 17872
2017-01-05 17:00:08,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1483603191971_0002_01_000001 transitioned from RUNNING to KILLING
2017-01-05 17:00:08,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1483603191971_0002_01_000001
2017-01-05 17:00:08,779 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM
2017-01-05 17:00:08,822 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1483603191971_0002_01_000001 is : 143
2017-01-05 17:00:08,825 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1483603191971_0002_01_000002 is : 143

Is the memory available on my pc not enough or are there any known issues which might lead to this?

Also, this doesn't occur every time I start a flink session.

Thanks
Sachin
Yury Ruchin
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Running into memory issues while running on Yarn

Yury Ruchin
Hi,

You containers got killed by YARN for exceeding virtual memory limits. For some reason your container intensively allocate virtual memory while having free physical memory.

There are some gotchas regarding such issue on CentOS, caused by OS-specific aggressive virtual memory allocation: [1], [2]. They disable YARN virtual memory checker to work around that.

Also in this mailing list people recently reported that high virtual memory consumption may be caused by some libraries.

Links:
[2] https://www.mapr.com/blog/best-practices-yarn-resource-management, section "3. Virtual/physical memory checker".

Regards,
Yury

2017-01-05 11:54 GMT+03:00 Sachin Goel <[hidden email]>:
Hey!

I'm running locally under this configuration(copied from nodemanager logs):
physical-memory=8192 virtual-memory=17204 virtual-cores=8

Before starting a flink deployment, memory usage stats show 3.7 GB used on system, indicating lots of free memory for flink containers.
However, after I submit using minimal resource requirements, ./yarn-session.sh -n 1 -tm 768, the cluster deploys successfully but then every application on system receives a sigterm and it basically kills the current user session, logging out of the system.

The job manager and task manager logs contain just the information that a SIGTERM was received and shut down gracefully.
All yarn and dfs process contain the log information showing the receipt of a sigterm.

Here's the relevant log from nodemanager:

2017-01-05 17:00:06,089 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1483603191971_0002_01_000002 transitioned from LOCALIZED to RUNNING
2017-01-05 17:00:06,092 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /opt/hadoop-2.7.3/tmp/nm-local-dir/usercache/kirk/appcache/application_1483603191971_0002/container_1483603191971_0002_01_000002/default_container_executor.sh]
2017-01-05 17:00:08,731 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1483603191971_0002_01_000002
2017-01-05 17:00:08,744 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 17872 for container-id container_1483603191971_0002_01_000001: 282.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used
2017-01-05 17:00:08,744 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_1483603191971_0002_01_000001 has processes older than 1 iteration running over the configured limit. Limit=2254857728, current usage = 2255896576
2017-01-05 17:00:08,745 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=17872,containerID=container_1483603191971_0002_01_000001] is running beyond virtual memory limits. Current usage: 282.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1483603191971_0002_01_000001 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 17872 17870 17872 17872 (bash) 0 0 21409792 812 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Xmx424M  -Dlog.file=/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnApplicationMasterRunner  1>/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.out 2>/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.err 
	|- 17879 17872 17872 17872 (java) 748 20 2234486784 71553 /usr/lib/jvm/java-8-openjdk-amd64//bin/java -Xmx424M -Dlog.file=/opt/hadoop-2.7.3/logs/userlogs/application_1483603191971_0002/container_1483603191971_0002_01_000001/jobmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnApplicationMasterRunner 

2017-01-05 17:00:08,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Removed ProcessTree with root 17872
2017-01-05 17:00:08,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1483603191971_0002_01_000001 transitioned from RUNNING to KILLING
2017-01-05 17:00:08,746 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1483603191971_0002_01_000001
2017-01-05 17:00:08,779 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM
2017-01-05 17:00:08,822 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1483603191971_0002_01_000001 is : 143
2017-01-05 17:00:08,825 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1483603191971_0002_01_000002 is : 143

Is the memory available on my pc not enough or are there any known issues which might lead to this?

Also, this doesn't occur every time I start a flink session.

Thanks
Sachin
Free forum by Nabble Edit this page