(DEPRECATED) Apache Flink User Mailing List archive.

question: jvm heap size per task?

classic Classic list List threaded Threaded
♦
♦
 3 messages Options
Li Zhao
Reply | Threaded
Open this post in threaded view
♦
♦
|

question: jvm heap size per task?

Li Zhao
Hi,

Greetings, hope this is the proper place to ask questions, apologize if not.

We have a shared flink cluster running with docker, want to set different heap size per task(some tasks require larger heap size, while most tasks only need a little), is it feasible?

I've gone through [1], [2] and [3], my current understanding is all task managers have the same heap size which is set by `taskmanager.heap.size` and is fixed ever since the task manager is started, and all tasks running in that task manager will share that heap size.

Am I understanding it correctly? And any approaches to our needs?

Thanks in advance!


Thanks,
Li
Xintong Song
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: question: jvm heap size per task?

Xintong Song
Hi Li,

It is true that currently all the task managers have the same heap size, and it's fixed ever since started. Unfortunately, your needs cannot be satisfied at the moment.

Heap size of task managers cannot be changed once started, because flink task managers run in JVMs and JVM does not support resizing after started.

However, there is an ongoing approach towards your needs. The community is working on fine-grained resource management, which in general allows specify per task/operator resource requirements and allocate task manager resources accordingly. You can refers to FLIP-53 [1] and FLIP-56 [2] for more details. Another related effort is pluggable slot manager [3], which allows having pluggable resource scheduling strategies such as launch task managers with customized resources according to the tasks' requirements.

Thank you~

Xintong Song



On Wed, Dec 25, 2019 at 4:18 PM Li Zhao <[hidden email]> wrote:
Hi,

Greetings, hope this is the proper place to ask questions, apologize if not.

We have a shared flink cluster running with docker, want to set different heap size per task(some tasks require larger heap size, while most tasks only need a little), is it feasible?

I've gone through [1], [2] and [3], my current understanding is all task managers have the same heap size which is set by `taskmanager.heap.size` and is fixed ever since the task manager is started, and all tasks running in that task manager will share that heap size.

Am I understanding it correctly? And any approaches to our needs?

Thanks in advance!


Thanks,
Li
Li Zhao
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: question: jvm heap size per task?

Li Zhao
Understood, thank you for the quick response!

Thanks,
Li

Xintong Song <[hidden email]> 于2019年12月25日周三 下午5:05写道：
Hi Li,

It is true that currently all the task managers have the same heap size, and it's fixed ever since started. Unfortunately, your needs cannot be satisfied at the moment.

Heap size of task managers cannot be changed once started, because flink task managers run in JVMs and JVM does not support resizing after started.

However, there is an ongoing approach towards your needs. The community is working on fine-grained resource management, which in general allows specify per task/operator resource requirements and allocate task manager resources accordingly. You can refers to FLIP-53 [1] and FLIP-56 [2] for more details. Another related effort is pluggable slot manager [3], which allows having pluggable resource scheduling strategies such as launch task managers with customized resources according to the tasks' requirements.

Thank you~

Xintong Song



On Wed, Dec 25, 2019 at 4:18 PM Li Zhao <[hidden email]> wrote:
Hi,

Greetings, hope this is the proper place to ask questions, apologize if not.

We have a shared flink cluster running with docker, want to set different heap size per task(some tasks require larger heap size, while most tasks only need a little), is it feasible?

I've gone through [1], [2] and [3], my current understanding is all task managers have the same heap size which is set by `taskmanager.heap.size` and is fixed ever since the task manager is started, and all tasks running in that task manager will share that heap size.

Am I understanding it correctly? And any approaches to our needs?

Thanks in advance!


Thanks,
Li
Free forum by Nabble Edit this page