yarn kill container due to running beyond physical memory limits [ How can i debug memory issue ]

[prateekarora]

Hi

I am running flink 1.0.2 with Yarn .

After running application for some time , Yarn kill my container due to running beyond physical memory limits .

how can i debug memory issue ?

below are the logs :

Container container_1463184272818_0165_01_000012 is completed with diagnostics: Container [pid=19349,containerID=container_1463184272818_0165_01_000012] is running beyond physical memory limits. Current usage: 6.0 GB of 6 GB physical memory used; 9.1 GB of 12.6 GB virtual memory used. Killing container.

Dump of the process-tree for container_1463184272818_0165_01_000012 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 19356 19349 19349 19349 (java) 39350 9110 9711140864 1581168 /usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m -XX:MaxDirectMemorySize=4608m -Djava.library.path=/home/nativelibraries/native_lib/ -Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
        |- 19349 19345 19349 19349 (bash) 0 0 11456512 359 /bin/bash -c /usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m -XX:MaxDirectMemorySize=4608m -Djava.library.path=/home/nativelibraries/native_lib/ -Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1> /var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.out 2> /var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.err

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

[Alexis Gendronneau]

Hello,

How many memory your yarn containers are configured to have ? This error may be due to running a flink on yarn cluster with more memory than you have in containers. Could you check it, and maybe set containers memory to a more suitable value ?

regards

2016-06-01 1:22 GMT+02:00 prateekarora <[hidden email]>:

Hi

I am running flink 1.0.2 with Yarn .

After running application for some time , Yarn kill my container due to
running beyond physical memory limits .

how can i debug memory issue ?

below are the logs :

Container container_1463184272818_0165_01_000012 is completed with
diagnostics: Container
[pid=19349,containerID=container_1463184272818_0165_01_000012] is running
beyond physical memory limits. Current usage: 6.0 GB of 6 GB physical memory
used; 9.1 GB of 12.6 GB virtual memory used. Killing container.

Dump of the process-tree for container_1463184272818_0165_01_000012 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 19356 19349 19349 19349 (java) 39350 9110 9711140864 1581168
/usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
-XX:MaxDirectMemorySize=4608m
-Djava.library.path=/home/nativelibraries/native_lib/
-Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log
-Dlogback.configurationFile=file:logback.xml
-Dlog4j.configuration=file:log4j.properties
org.apache.flink.yarn.YarnTaskManagerRunner --configDir .
        |- 19349 19345 19349 19349 (bash) 0 0 11456512 359 /bin/bash -c
/usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
-XX:MaxDirectMemorySize=4608m
-Djava.library.path=/home/nativelibraries/native_lib/
-Dlog.file=/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.log
-Dlogback.configurationFile=file:logback.xml
-Dlog4j.configuration=file:log4j.properties
org.apache.flink.yarn.YarnTaskManagerRunner --configDir . 1>
/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.out
2>
/var/log/hadoop-yarn/container/application_1463184272818_0165/container_1463184272818_0165_01_000012/taskmanager.err

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/yarn-kill-container-due-to-running-beyond-physical-memory-limits-How-can-i-debug-memory-issue-tp7296.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

--

[prateekarora]

Hi

Thanks for the reply

i have 6 node yarn cluster with total 107.66 GB Memory and 48 vcore .

configuration :
    5 Node :
            configure each  Node with 19.53 GiB  ( yarn.nodemanager.resource.memory-mb = 19.53 GB)

    1 Node :
           configure Node with 10 GiB  ( yarn.nodemanager.resource.memory-mb = 10 GB)


    Total : around 107.66 GB


currently i am running my flink application using below commnad :

            flink run -m yarn-cluster -yn 15 -ytm  6144 -ys 1  <application_jar>

     if i tried to run my application using  below configuration then also facing same issue.
         
           flink run -m yarn-cluster -yn 15 -ytm  4096 -ys 1  <application_jar>


Regards
Prateek

[snntr]

Hi Prateek,

did you change "yarn.heap-cutoff-ratio" or "yarn.heap-cutoff-ratio.min"
[1]?

Cheers,

Konstantin

[1]
https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#yarn


On 01.06.2016 17:46, prateekarora wrote:

> Hi
>
> Thanks for the reply
>
> i have 6 node yarn cluster with total 107.66 GB Memory and 48 vcore .
>
> configuration :
>     5 Node :
>             configure each  Node with 19.53 GiB  (
> yarn.nodemanager.resource.memory-mb = 19.53 GB)
>
>     1 Node :
>            configure Node with 10 GiB  ( yarn.nodemanager.resource.memory-mb
> = 10 GB)
>
>
>     Total : around 107.66 GB
>
>
> currently i am running my flink application using below commnad :
>
>             flink run -m yarn-cluster -yn 15 -ytm  6144 -ys 1
> <application_jar>
>
>      if i tried to run my application using  below configuration then also
> facing same issue.
>          
>            flink run -m yarn-cluster -yn 15 -ytm  4096 -ys 1
> <application_jar>
>
>
> Regards
> Prateek
>
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/yarn-kill-container-due-to-running-beyond-physical-memory-limits-How-can-i-debug-memory-issue-tp7296p7317.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>

--
Konstantin Knauf * [hidden email] * [phone]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

[prateekarora]

Hi

I have not changed any configuration "yarn.heap-cutoff-ratio" or "yarn.heap-cutoff-ratio.min" .

As per log  flink assign 4608 M out of 6 GB .  i thought  configuration working fine .
/usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
-XX:MaxDirectMemorySize=4608m


Regards
Prateek
-

[Alexis Gendronneau]

thats kinda strange. Have you tried setting minimum allocations for containers to a bigger size too ?

2016-06-02 0:04 GMT+02:00 prateekarora <[hidden email]>:

Hi

I have not changed any configuration "yarn.heap-cutoff-ratio" or
"yarn.heap-cutoff-ratio.min" .

As per log  flink assign 4608 M out of 6 GB .  i thought  configuration
working fine .
/usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xms4608m -Xmx4608m
-XX:MaxDirectMemorySize=4608m


Regards
Prateek
-



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/yarn-kill-container-due-to-running-beyond-physical-memory-limits-How-can-i-debug-memory-issue-tp7296p7325.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

--