Question regarding cpu limit config in Flink standalone mode

[Fan Xie]

Hi Flink Community,

Recently I am working on an auto-scaling project that needs to dynamically adjust the cpu config of Flink standalone jobs . Our jobs will be running on standalone mode in a k8s cluster. After going through the configuration doc: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/, I can't find a config that can directly control the cpu of a standalone flink job. I can only see kubernetes.taskmanager.cpu, but looks like this config is only useful in native k8s mode. I also notice another config: taskmanager.numberOfTaskSlots that can control the cpu config in an indirect way. Is there any reason why we can't config the cpu for a standalone job directly? 

Thanks for answering my question. 

Best,
Fan 

[Xintong Song]

Hi Fan,

For a java application, you cannot specify how many cpu a process should use. The JVM process will always try to use as much cpu time as it needs. The limitation can only come from external: hardware limit, OS scheduling, cgroups, etc.

On Kubernetes, it is the pod's resource specifications that decide how many cpu resources a Flink JM/TM can use.

- For the standalone kubernetes deployment, you can specify the pods' resources in your yaml files.

- For the native kubernetes deployment, TM pods are requested by Flink's ResourceManager. Thus, the configuration option `kubernets.taskmanager.cpu` controls the cpu resource of pods Flink requests from Kubernetes.

On Fri, May 7, 2021 at 10:35 AM Fan Xie <[hidden email]> wrote:

Hi Flink Community,

Recently I am working on an auto-scaling project that needs to dynamically adjust the cpu config of Flink standalone jobs . Our jobs will be running on standalone mode in a k8s cluster. After going through the configuration doc: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/, I can't find a config that can directly control the cpu of a standalone flink job. I can only see kubernetes.taskmanager.cpu, but looks like this config is only useful in native k8s mode. I also notice another config: taskmanager.numberOfTaskSlots that can control the cpu config in an indirect way. Is there any reason why we can't config the cpu for a standalone job directly? 

Thanks for answering my question. 

Best,
Fan 

[Fan Xie]

Hi Xintong,

Thanks for answering my question! After discussing with my teammates, we decide to rely on k8s pod and an external control plane to restrict the CPU usage of a job.

Thanks again for your help!

Best,

Fan

---

From: Xintong Song <[hidden email]>
Sent: Thursday, May 6, 2021 8:39 PM
To: [hidden email] <[hidden email]>
Subject: Re: Question regarding cpu limit config in Flink standalone mode

 

Hi Fan,

For a java application, you cannot specify how many cpu a process should use. The JVM process will always try to use as much cpu time as it needs. The limitation can only come from external: hardware limit, OS scheduling, cgroups, etc.

On Kubernetes, it is the pod's resource specifications that decide how many cpu resources a Flink JM/TM can use.

- For the standalone kubernetes deployment, you can specify the pods' resources in your yaml files.

- For the native kubernetes deployment, TM pods are requested by Flink's ResourceManager. Thus, the configuration option `kubernets.taskmanager.cpu` controls the cpu resource of pods Flink requests from Kubernetes.

On Fri, May 7, 2021 at 10:35 AM Fan Xie <[hidden email]> wrote:

Hi Flink Community,

Recently I am working on an auto-scaling project that needs to dynamically adjust the cpu config of Flink standalone jobs . Our jobs will be running on standalone mode in a k8s cluster. After going through the configuration doc: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/, I can't find a config that can directly control the cpu of a standalone flink job. I can only see kubernetes.taskmanager.cpu, but looks like this config is only useful in native k8s mode. I also notice another config: taskmanager.numberOfTaskSlots that can control the cpu config in an indirect way. Is there any reason why we can't config the cpu for a standalone job directly? 

Thanks for answering my question. 

Best,
Fan