Flink 1.10.1 not using FLINK_TM_HEAP for TaskManager JVM Heap size correctly?

[Li Peng-2]

Hey folks, we recently migrated from Flink 1.9.x to 1.10.1, and we noticed some wonky behavior in how JVM is configured:

1. We Add FLINK_JM_HEAP=5000m and FLINK_TM_HEAP=1400m variables to the environment
2. The JobManager allocates the right heap size as expected

3. However, the TaskManager (started via taskmanager.sh), logs this instead:

 - 'taskmanager.memory.flink.size' is not specified, use the configured deprecated task manager heap value (1.367gb (1468006400 bytes)) for it.
 - The derived from fraction jvm overhead memory (184.000mb (192937987 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
BASH_JAVA_UTILS_EXEC_RESULT:-Xmx599785462 -Xms599785462 -XX:MaxDirectMemorySize=281018370 -XX:MaxMetaspaceSize=268435456

So the logs say it will use the configured 1400m as expected, but for some reason it picks 599785462 as the heap size instead (TaskManagerRunner logs that Maximum heap size is 572 MiBytes, so it's verified that the 1400m value is not used)? 

Anyone know if I'm missing a setting here or something?

Thanks,

Li

[Xintong Song]

Hi Li,

FLINK_TM_HEAP corresponds to the legacy configuration option "taskmanager.heap.size". It is supported for backwards compatibility. I strongly recommend you to use "taskmanager.memory.flink.size" or "taskmanager.memory.process.size" instead, which can be passed either in "flink-conf.yaml" or through "-D" options.

This configuration option, while confusingly named with "heap", is actually specifying the total memory of task manager, including the off-heap memory components. 

This is also documented as follows[1] (for the configuration option but not for the environment variable)

The previous options which were responsible for the total memory used by Flink are taskmanager.heap.size or taskmanager.heap.mb. Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_migration.html#total-memory-previously-heap-memory

On Fri, Jun 12, 2020 at 4:09 PM Li Peng <[hidden email]> wrote:

Hey folks, we recently migrated from Flink 1.9.x to 1.10.1, and we noticed some wonky behavior in how JVM is configured:

1. We Add FLINK_JM_HEAP=5000m and FLINK_TM_HEAP=1400m variables to the environment
2. The JobManager allocates the right heap size as expected

3. However, the TaskManager (started via taskmanager.sh), logs this instead:

 - 'taskmanager.memory.flink.size' is not specified, use the configured deprecated task manager heap value (1.367gb (1468006400 bytes)) for it.
 - The derived from fraction jvm overhead memory (184.000mb (192937987 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
BASH_JAVA_UTILS_EXEC_RESULT:-Xmx599785462 -Xms599785462 -XX:MaxDirectMemorySize=281018370 -XX:MaxMetaspaceSize=268435456

So the logs say it will use the configured 1400m as expected, but for some reason it picks 599785462 as the heap size instead (TaskManagerRunner logs that Maximum heap size is 572 MiBytes, so it's verified that the 1400m value is not used)? 

Anyone know if I'm missing a setting here or something?

Thanks,

Li

[Xintong Song]

To be more specific, your 1400m total memory should also consists of:

• 572MB heap memory (-Xmx & -Xms)
• 268MB direct memory (-XX:MaxDirectMemorySize)
• 560MB managed memory (native memory, calculated as 1400m * managedMemoryFraction, the fraction is by default 0.4)

On Fri, Jun 12, 2020 at 4:27 PM Xintong Song <[hidden email]> wrote:

Hi Li,

FLINK_TM_HEAP corresponds to the legacy configuration option "taskmanager.heap.size". It is supported for backwards compatibility. I strongly recommend you to use "taskmanager.memory.flink.size" or "taskmanager.memory.process.size" instead, which can be passed either in "flink-conf.yaml" or through "-D" options.

This configuration option, while confusingly named with "heap", is actually specifying the total memory of task manager, including the off-heap memory components. 

This is also documented as follows[1] (for the configuration option but not for the environment variable)

The previous options which were responsible for the total memory used by Flink are taskmanager.heap.size or taskmanager.heap.mb. Despite their naming, they included not only JVM heap but also other off-heap memory components. The options have been deprecated.

Thank you~

Xintong Song

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_migration.html#total-memory-previously-heap-memory

On Fri, Jun 12, 2020 at 4:09 PM Li Peng <[hidden email]> wrote:

Hey folks, we recently migrated from Flink 1.9.x to 1.10.1, and we noticed some wonky behavior in how JVM is configured:

1. We Add FLINK_JM_HEAP=5000m and FLINK_TM_HEAP=1400m variables to the environment
2. The JobManager allocates the right heap size as expected

3. However, the TaskManager (started via taskmanager.sh), logs this instead:

 - 'taskmanager.memory.flink.size' is not specified, use the configured deprecated task manager heap value (1.367gb (1468006400 bytes)) for it.
 - The derived from fraction jvm overhead memory (184.000mb (192937987 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
BASH_JAVA_UTILS_EXEC_RESULT:-Xmx599785462 -Xms599785462 -XX:MaxDirectMemorySize=281018370 -XX:MaxMetaspaceSize=268435456

So the logs say it will use the configured 1400m as expected, but for some reason it picks 599785462 as the heap size instead (TaskManagerRunner logs that Maximum heap size is 572 MiBytes, so it's verified that the 1400m value is not used)? 

Anyone know if I'm missing a setting here or something?

Thanks,

Li