Flink memory usage monitoring

[Rajesh Payyappilly Jose]

Classification: Internal

Hi,

 

Environment - Flink 1.11 on K8s

 

Is there a way to monitor the usage of managed memory, off-heap memory and network memory?

 

-Rajesh

 

::DISCLAIMER::

---

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

---

[Matthias]

I missed adding the mailing list in my previous email.

---------- Forwarded message ---------
From: Matthias Pohl <[hidden email]>
Date: Tue, Oct 27, 2020 at 12:39 PM
Subject: Re: Flink memory usage monitoring
To: Rajesh Payyappilly Jose <[hidden email]>

Hi Rajesh,

thanks for reaching out to us. We worked on providing metrics for managed memory and network memory as part of FLIP-102 [1]. It looks like these features are going to be added to the upcoming release of Flink 1.12.

We decided to not include off-heap memory as it is not necessarily under control of Flink (e.g. user code can allocate native memory and Flink wouldn't be aware of it). Hence, providing numbers for off-heap memory usage might be misleading. There will be a metric to monitor the Metaspace usage, though.

Best,
Matthias

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-102%3A+Add+More+Metrics+to+TaskManager

On Tue, Oct 20, 2020 at 8:23 PM Rajesh Payyappilly Jose <[hidden email]> wrote:

Classification: Internal

Hi,

 

Environment - Flink 1.11 on K8s

 

Is there a way to monitor the usage of managed memory, off-heap memory and network memory?

 

-Rajesh

 

::DISCLAIMER::

---

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.