questions about yarn container（taskmanager）memory allocation

[XiangWei Huang]

Hi all,

    Currently i am running my flink application in yarn session mode and using below commnad :

bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096

when taskmanager complete to started,i found the container launching command is :

bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ......

from flink source code,i'v learned how direct memory size and heap memory size calculated :

container total memory: 4096M

network memory fraction: 0.1

cut off memory fraction: 0.25

networkMemory = 4096*0.1 = 409.6

cut off memory = (4096 - 409.6) * 0.25 = 921.6

directMemorySize = 409.6 + 921.6 = 1331.2

heapsize = 4096 - 1331.2 = 2764.8

Below is the environment i'm using to run yarn and flink:

  jdk version: 1.8

 flink version: 1.6.1

OS: centos7

 

there are two questions about memory allocation i want to ask:

1. Since the jdk version i an using is 1.8,is it necessary to consider the metaspace memory into calculation.According to the current way to calculate memory size（without metaspace memory） may cause 

container running beyond physical memory limit.

2. Is the native memory that rocksdb using part of direct memory（limit by jvm parameter  MaxDirectMemorySize）,if not how do i control the size it used.

[Xintong Song]

Hi XiangWei,

Thank you for the inputs. I agree with you that it is possible that containers may use extra memory in 1.8. As for native memory, it is memory used by JVM and other processes outside JVM. So it's not limited by MaxDirectMemorySize.

The community is working on a refactoring plan about resource management of Flink. AFAIK, the native memory is on the table, but I'm not sure whether the metaspace memory is considered. I think we should create a jira issue on this. 

On Mon, May 20, 2019 at 4:47 PM XiangWei Huang <[hidden email]> wrote:

Hi all,

    Currently i am running my flink application in yarn session mode and using below commnad :

bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096

when taskmanager complete to started,i found the container launching command is :

bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ......

from flink source code,i'v learned how direct memory size and heap memory size calculated :

container total memory: 4096M

network memory fraction: 0.1

cut off memory fraction: 0.25

networkMemory = 4096*0.1 = 409.6

cut off memory = (4096 - 409.6) * 0.25 = 921.6

directMemorySize = 409.6 + 921.6 = 1331.2

heapsize = 4096 - 1331.2 = 2764.8

Below is the environment i'm using to run yarn and flink:

  jdk version: 1.8

 flink version: 1.6.1

OS: centos7

 

there are two questions about memory allocation i want to ask:

1. Since the jdk version i an using is 1.8,is it necessary to consider the metaspace memory into calculation.According to the current way to calculate memory size（without metaspace memory） may cause 

container running beyond physical memory limit.

2. Is the native memory that rocksdb using part of direct memory（limit by jvm parameter  MaxDirectMemorySize）,if not how do i control the size it used.