Setting job/task manager memory management in kubernetes

[Sakshi Bansal]

Hello,

I am trying to set the heap size of job and task manager when deploying the job in kubernetes. I have set the jobmanager.heap.size and taskmanager.heap.size. However, the custom values are not being used and it is creating its own values and starting the job. How can I set custom values?

--

[Yangze Guo]

Hi, Sakshi

Could you provide more information about:
- What is the Flink version you are using? "taskmanager.heap.size" is
deprecated since 1.10[1].
- How do you deploy the cluster? In the approach of native k8s[2] or
the standalone k8s[3]?

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_migration.html
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/cluster_setup.html
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html

Best,
Yangze Guo

On Mon, Aug 24, 2020 at 6:31 PM Sakshi Bansal <[hidden email]> wrote:
>
> Hello,
>
> I am trying to set the heap size of job and task manager when deploying the job in kubernetes. I have set the jobmanager.heap.size and taskmanager.heap.size. However, the custom values are not being used and it is creating its own values and starting the job. How can I set custom values?
>
> --
> Thanks and Regards
> Sakshi Bansal

[Yangze Guo]

Hi,

You need to define them in "flink-configuration-configmap.yaml".
Please also make sure you've created the config map by executing
"kubectl create -f flink-configuration-configmap.yaml".

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/kubernetes.html

Best,
Yangze Guo

On Mon, Aug 24, 2020 at 9:33 PM Sakshi Bansal <[hidden email]> wrote:

>
> The flink version is 1.9 and it is a standalone k8s
>
> On Mon 24 Aug, 2020, 17:17 Yangze Guo, <[hidden email]> wrote:
>>
>> Hi, Sakshi
>>
>> Could you provide more information about:
>> - What is the Flink version you are using? "taskmanager.heap.size" is
>> deprecated since 1.10[1].
>> - How do you deploy the cluster? In the approach of native k8s[2] or
>> the standalone k8s[3]?
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_migration.html
>> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/cluster_setup.html
>> [3] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/kubernetes.html
>>
>> Best,
>> Yangze Guo
>>
>> On Mon, Aug 24, 2020 at 6:31 PM Sakshi Bansal <[hidden email]> wrote:
>> >
>> > Hello,
>> >
>> > I am trying to set the heap size of job and task manager when deploying the job in kubernetes. I have set the jobmanager.heap.size and taskmanager.heap.size. However, the custom values are not being used and it is creating its own values and starting the job. How can I set custom values?
>> >
>> > --
>> > Thanks and Regards
>> > Sakshi Bansal

[Alexey Trenikhun]

Hello,

What version of Flink do you use? If you use 1.10+ please check [1] (different properties names)

[1] - https://ci.apache.org/projects/flink/flink-docs-stable/ops/memory/mem_setup.html

Thanks,

Alexey

---

From: Sakshi Bansal <[hidden email]>
Sent: Monday, August 24, 2020 3:30 AM
To: [hidden email] <[hidden email]>
Subject: Setting job/task manager memory management in kubernetes

 

Hello,

I am trying to set the heap size of job and task manager when deploying the job in kubernetes. I have set the jobmanager.heap.size and taskmanager.heap.size. However, the custom values are not being used and it is creating its own values and starting the job. How can I set custom values?

--