Flink on YARN: TaskManager heap auto-sizing?

[Dylan Adams]

In the Configuration section of the docs, the description for "taskmanager.heap.size" contains: "On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value."

Does that functionality exist?

I don't see any documented method to specify the YARN container size for the TaskManagers, nor could I find any logic in the Flink YARN integration code that seemed to implement that behavior.

My understanding is that you need to manually calculate and specify taskmanager.heap.size (and jobmanager.heap.size) based on your YARN setup.

Thanks,

Dylan

[Till Rohrmann]

Hi Dylan,

the container's memory will be calculated here [1]. In the case of Yarn, the user specifies the container memory size and based on this Flink calculates with how much heap memory the JVM is started (container memory size - off heap memory - cut off memory).

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParameters.java#L160

Cheers,

Till

On Tue, May 7, 2019 at 3:29 PM Dylan Adams <[hidden email]> wrote:

In the Configuration section of the docs, the description for "taskmanager.heap.size" contains: "On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value."

Does that functionality exist?

I don't see any documented method to specify the YARN container size for the TaskManagers, nor could I find any logic in the Flink YARN integration code that seemed to implement that behavior.

My understanding is that you need to manually calculate and specify taskmanager.heap.size (and jobmanager.heap.size) based on your YARN setup.

Thanks,

Dylan

[Dylan Adams]

Till,

Thanks for the pointer to the code.

Regards,

Dylan

On Wed, May 8, 2019 at 11:18 AM Till Rohrmann <[hidden email]> wrote:

Hi Dylan,

the container's memory will be calculated here [1]. In the case of Yarn, the user specifies the container memory size and based on this Flink calculates with how much heap memory the JVM is started (container memory size - off heap memory - cut off memory).

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParameters.java#L160

Cheers,

Till

On Tue, May 7, 2019 at 3:29 PM Dylan Adams <[hidden email]> wrote:

In the Configuration section of the docs, the description for "taskmanager.heap.size" contains: "On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value."

Does that functionality exist?

I don't see any documented method to specify the YARN container size for the TaskManagers, nor could I find any logic in the Flink YARN integration code that seemed to implement that behavior.

My understanding is that you need to manually calculate and specify taskmanager.heap.size (and jobmanager.heap.size) based on your YARN setup.

Thanks,

Dylan