Best way set max heap size via env variables or program arguments?

[Li Peng-2]

Hey folks, we've been running a k8 flink application, using the taskmanager.sh script and passing in the -Djobmanager.heap.size=9000m and -Dtaskmanager.heap.size=7000m as options to the script. I noticed from the logs, that the Maximum heap size logged completely ignores these arguments, and just sets the heap to the default of 922M. 

I tested setting the taskmanager.heap.size and jobmanager.heap.size manually in flink-conf.yaml, and it does work as expected (minus the heap being set a little lower than configured). But since we want the application to pick up different memory settings based on the environment (local/staging/prod/etc), setting it in flink-conf isn't ideal.

So my questions are:

1. Is there actually a way to pass in the heap size via arguments to taskmanager.sh? Is passing -Dtaskmanager.heap.size supposed to work?

2.  If not, is there a recommended way to set the heap size by environment, like environmental variables?

3. Also, the maximum heap size logged and -Xms and -Xmx is always a little smaller than the configured size (i.e. configuring 3000m results 2700m in the jvm arguments, 1024m results in 922), why is that? 

Thanks, and happy new year!
Li

[Xintong Song]

Hi Li,

Regarding your questions:

1. Is there actually a way to pass in the heap size via arguments to taskmanager.sh? Is passing -Dtaskmanager.heap.size supposed to work?

No, '-Dtaskmanager.heap.size' is not supposed to work. The '-D' configurations are only parsed after the JVM is started, while 'taskmanager.heap.size' will be used for starting the JVM.

2.  If not, is there a recommended way to set the heap size by environment, like environmental variables?

Yes, these is an equivalent environment variable FLINK_TM_HEAP. 

On Wed, Jan 1, 2020 at 3:10 AM Li Peng <[hidden email]> wrote:

Hey folks, we've been running a k8 flink application, using the taskmanager.sh script and passing in the -Djobmanager.heap.size=9000m and -Dtaskmanager.heap.size=7000m as options to the script. I noticed from the logs, that the Maximum heap size logged completely ignores these arguments, and just sets the heap to the default of 922M. 

I tested setting the taskmanager.heap.size and jobmanager.heap.size manually in flink-conf.yaml, and it does work as expected (minus the heap being set a little lower than configured). But since we want the application to pick up different memory settings based on the environment (local/staging/prod/etc), setting it in flink-conf isn't ideal.

So my questions are:

1. Is there actually a way to pass in the heap size via arguments to taskmanager.sh? Is passing -Dtaskmanager.heap.size supposed to work?

2.  If not, is there a recommended way to set the heap size by environment, like environmental variables?

3. Also, the maximum heap size logged and -Xms and -Xmx is always a little smaller than the configured size (i.e. configuring 3000m results 2700m in the jvm arguments, 1024m results in 922), why is that? 

Thanks, and happy new year!
Li

[Li Peng-2]

Awesome, thanks!

On Wed, Jan 1, 2020 at 6:17 PM Xintong Song <[hidden email]> wrote:

Hi Li,

Regarding your questions:

1. Is there actually a way to pass in the heap size via arguments to taskmanager.sh? Is passing -Dtaskmanager.heap.size supposed to work?

No, '-Dtaskmanager.heap.size' is not supposed to work. The '-D' configurations are only parsed after the JVM is started, while 'taskmanager.heap.size' will be used for starting the JVM.

2.  If not, is there a recommended way to set the heap size by environment, like environmental variables?

Yes, these is an equivalent environment variable FLINK_TM_HEAP. 

3. Also, the maximum heap size logged and -Xms and -Xmx is always a little smaller than the configured size (i.e. configuring 3000m results 2700m in the jvm arguments, 1024m results in 922), why is that? 

The configuration key 'taskmanager.heap.size' is a bit inaccurate. The config option actually also account for some off-heap memory, such as network direct buffers and off-heap managed memory (if used). That's way you see the java heap size is always slightly smaller than the configured 'taskmanager.heap.size'.

 

Thank you~

Xintong Song

On Wed, Jan 1, 2020 at 3:10 AM Li Peng <[hidden email]> wrote:

Hey folks, we've been running a k8 flink application, using the taskmanager.sh script and passing in the -Djobmanager.heap.size=9000m and -Dtaskmanager.heap.size=7000m as options to the script. I noticed from the logs, that the Maximum heap size logged completely ignores these arguments, and just sets the heap to the default of 922M. 

I tested setting the taskmanager.heap.size and jobmanager.heap.size manually in flink-conf.yaml, and it does work as expected (minus the heap being set a little lower than configured). But since we want the application to pick up different memory settings based on the environment (local/staging/prod/etc), setting it in flink-conf isn't ideal.

So my questions are:

1. Is there actually a way to pass in the heap size via arguments to taskmanager.sh? Is passing -Dtaskmanager.heap.size supposed to work?

2.  If not, is there a recommended way to set the heap size by environment, like environmental variables?

3. Also, the maximum heap size logged and -Xms and -Xmx is always a little smaller than the configured size (i.e. configuring 3000m results 2700m in the jvm arguments, 1024m results in 922), why is that? 

Thanks, and happy new year!
Li