Implications of taskmanager.memory.process.size

[Vishal Santoshi]

Hello folks, 

                  As established https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration , I set the taskmanager.memory.process.size and taskmanager.memory.task.off-heap.size in my flink-conf.yaml and I see the 2 properties being pulled in.

 - Loading configuration property: taskmanager.memory.process.size, 8000m

 - Loading configuration property: taskmanager.memory.task.off-heap.size, 1024m

I am not sure how the -Xmx and -Xms are calculated but I see 

Starting taskexecutor as a console application on host kafka-to-hdfs-taskmanager-dev-8597c78d9c-59dqw.

VM settings:

    Min. Heap Size: 2.27G

    Max. Heap Size: 2.27G

    Using VM: OpenJDK 64-Bit Server VM

What gives ? 

I am looking through the scripts and am not sure I see any calculations based on taskmanager.memory.process.size

[Vishal Santoshi]

ager.memory.process.size	(none)	MemorySize	Total Process Memory size for the TaskExecutors. This includes all the memory that a TaskExecutor consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead. On containerized setups, this should be set to the container memory. See also 'taskmanager.memory.flink.size' for total Flink memory size configuration.

The Total Flink Memory consists of off heap memory governed by fraction. In pure streaming case ( and non ROCKSDB state case )

This is the size of off-heap memory managed by the memory manager, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend. Memory consumers can either allocate memory from the memory manager in the form of MemorySegments, or reserve bytes from the memory manager and keep their memory usage within that boundary.

Is not used AFAIK . May be reduce the fraction to 0 ? We do not  use offline heap ( aka batch jobs ) on our cluster ?

Any help will be appreciated.

On Thu, Jul 9, 2020 at 9:25 AM Vishal Santoshi <[hidden email]> wrote:

Hello folks, 

                  As established https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration , I set the taskmanager.memory.process.size and taskmanager.memory.task.off-heap.size in my flink-conf.yaml and I see the 2 properties being pulled in.

 - Loading configuration property: taskmanager.memory.process.size, 8000m

 - Loading configuration property: taskmanager.memory.task.off-heap.size, 1024m

I am not sure how the -Xmx and -Xms are calculated but I see 

Starting taskexecutor as a console application on host kafka-to-hdfs-taskmanager-dev-8597c78d9c-59dqw.

VM settings:

    Min. Heap Size: 2.27G

    Max. Heap Size: 2.27G

    Using VM: OpenJDK 64-Bit Server VM

What gives ? 

I am looking through the scripts and am not sure I see any calculations based on taskmanager.memory.process.size

[Xintong Song]

Hi Vishal,

If you have streaming jobs only and do not use RocksDB, you can tune the fraction (taskmanager.memory.managed.fraction) to 0. In this way, no more off-heap managed memory will be reserved for the user code execution.

Please be aware that this does not mean the JVM heap will get all of the `process.size`. The Flink framework will still use some off-heap memory, for purposes like network buffering and JVM overhead.

On Thu, Jul 9, 2020 at 10:57 PM Vishal Santoshi <[hidden email]> wrote:

ager.memory.process.size	(none)	MemorySize	Total Process Memory size for the TaskExecutors. This includes all the memory that a TaskExecutor consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead. On containerized setups, this should be set to the container memory. See also 'taskmanager.memory.flink.size' for total Flink memory size configuration.

The Total Flink Memory consists of off heap memory governed by fraction. In pure streaming case ( and non ROCKSDB state case )

This is the size of off-heap memory managed by the memory manager, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend. Memory consumers can either allocate memory from the memory manager in the form of MemorySegments, or reserve bytes from the memory manager and keep their memory usage within that boundary.

Is not used AFAIK . May be reduce the fraction to 0 ? We do not  use offline heap ( aka batch jobs ) on our cluster ?

Any help will be appreciated.

On Thu, Jul 9, 2020 at 9:25 AM Vishal Santoshi <[hidden email]> wrote:

Hello folks, 

                  As established https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration , I set the taskmanager.memory.process.size and taskmanager.memory.task.off-heap.size in my flink-conf.yaml and I see the 2 properties being pulled in.

 - Loading configuration property: taskmanager.memory.process.size, 8000m

 - Loading configuration property: taskmanager.memory.task.off-heap.size, 1024m

I am not sure how the -Xmx and -Xms are calculated but I see 

Starting taskexecutor as a console application on host kafka-to-hdfs-taskmanager-dev-8597c78d9c-59dqw.

VM settings:

    Min. Heap Size: 2.27G

    Max. Heap Size: 2.27G

    Using VM: OpenJDK 64-Bit Server VM

What gives ? 

I am looking through the scripts and am not sure I see any calculations based on taskmanager.memory.process.size

[Vishal Santoshi]

Got it. That was what I always thought but needed to be sure. Thank you for confirming....

On Thu, Jul 9, 2020 at 9:39 PM Xintong Song <[hidden email]> wrote:

Hi Vishal,

If you have streaming jobs only and do not use RocksDB, you can tune the fraction (taskmanager.memory.managed.fraction) to 0. In this way, no more off-heap managed memory will be reserved for the user code execution.

Please be aware that this does not mean the JVM heap will get all of the `process.size`. The Flink framework will still use some off-heap memory, for purposes like network buffering and JVM overhead.

Thank you~

Xintong Song

On Thu, Jul 9, 2020 at 10:57 PM Vishal Santoshi <[hidden email]> wrote:

ager.memory.process.size	(none)	MemorySize	Total Process Memory size for the TaskExecutors. This includes all the memory that a TaskExecutor consumes, consisting of Total Flink Memory, JVM Metaspace, and JVM Overhead. On containerized setups, this should be set to the container memory. See also 'taskmanager.memory.flink.size' for total Flink memory size configuration.

The Total Flink Memory consists of off heap memory governed by fraction. In pure streaming case ( and non ROCKSDB state case )

This is the size of off-heap memory managed by the memory manager, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend. Memory consumers can either allocate memory from the memory manager in the form of MemorySegments, or reserve bytes from the memory manager and keep their memory usage within that boundary.

Is not used AFAIK . May be reduce the fraction to 0 ? We do not  use offline heap ( aka batch jobs ) on our cluster ?

Any help will be appreciated.

On Thu, Jul 9, 2020 at 9:25 AM Vishal Santoshi <[hidden email]> wrote:

Hello folks, 

                  As established https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration , I set the taskmanager.memory.process.size and taskmanager.memory.task.off-heap.size in my flink-conf.yaml and I see the 2 properties being pulled in.

 - Loading configuration property: taskmanager.memory.process.size, 8000m

 - Loading configuration property: taskmanager.memory.task.off-heap.size, 1024m

I am not sure how the -Xmx and -Xms are calculated but I see 

Starting taskexecutor as a console application on host kafka-to-hdfs-taskmanager-dev-8597c78d9c-59dqw.

VM settings:

    Min. Heap Size: 2.27G

    Max. Heap Size: 2.27G

    Using VM: OpenJDK 64-Bit Server VM

What gives ? 

I am looking through the scripts and am not sure I see any calculations based on taskmanager.memory.process.size