a couple of memory questions

[Colletta, Edward]

Using Flink 1.9.2 with FsStateBackend, Session cluster.

 

1. Does heap state get cleaned up when a job is cancelled?   

We have jobs that we run on a daily basis.  We start each morning and cancel each evening.  We noticed that the process size does not seem to shrink.  We are looking at the resident size of the process with ps and also the USED column for Heap on the taskmanager page of the flink dashboard.

2. How can I examine the usage of Flink Managed Memory?

 The configuration documentation seems to indicate this is used for batch jobs, and we are only using the Streaming API.   I reduced taskmanager.memory.fraction to 0.3, but I think this is still reserving too much memory to an area we will not be using. 

  

[Matthias]

Hello Edward,

please find my answers within your message below:

On Wed, Nov 4, 2020 at 1:35 PM Colletta, Edward <[hidden email]> wrote:

Using Flink 1.9.2 with FsStateBackend, Session cluster.

 

1. Does heap state get cleaned up when a job is cancelled?   

We have jobs that we run on a daily basis.  We start each morning and cancel each evening.  We noticed that the process size does not seem to shrink.  We are looking at the resident size of the process with ps and also the USED column for Heap on the taskmanager page of the flink dashboard.

There is no explicit cleanup happening on the Flink side. The heap should be cleaned up when GC kicks in. 

2. How can I examine the usage of Flink Managed Memory?

 The configuration documentation seems to indicate this is used for batch jobs, and we are only using the Streaming API.   I reduced taskmanager.memory.fraction to 0.3, but I think this is still reserving too much memory to an area we will not be using. 

Unfortunately, I don't know of any way to monitor the managed memory for Flink 1.9.2 as is. We're going to introduce new metrics for managed memory [1], network memory [2] and metaspace [3] in the upcoming release of Flink 1.12.0. This should make it easier to monitor these memory pools.

I hope that helps a bit.

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-14406

[2] https://issues.apache.org/jira/browse/FLINK-14422

[3] https://issues.apache.org/jira/browse/FLINK-19617

[Colletta, Edward]

Thanks you for the response.   We do see the heap actually shrink after starting new jobs. 

 

 

From: Matthias Pohl <[hidden email]>
Sent: Thursday, November 5, 2020 8:20 AM
To: Colletta, Edward <[hidden email]>
Cc: [hidden email]
Subject: Re: a couple of memory questions

 

This email is from an external source - exercise caution regarding links and attachments.

 

Hello Edward,

please find my answers within your message below:

 

On Wed, Nov 4, 2020 at 1:35 PM Colletta, Edward <[hidden email]> wrote:

Using Flink 1.9.2 with FsStateBackend, Session cluster.

 

1. Does heap state get cleaned up when a job is cancelled?   

We have jobs that we run on a daily basis.  We start each morning and cancel each evening.  We noticed that the process size does not seem to shrink.  We are looking at the resident size of the process with ps and also the USED column for Heap on the taskmanager page of the flink dashboard.

There is no explicit cleanup happening on the Flink side. The heap should be cleaned up when GC kicks in. 

2. How can I examine the usage of Flink Managed Memory?

 The configuration documentation seems to indicate this is used for batch jobs, and we are only using the Streaming API.   I reduced taskmanager.memory.fraction to 0.3, but I think this is still reserving too much memory to an area we will not be using. 

Unfortunately, I don't know of any way to monitor the managed memory for Flink 1.9.2 as is. We're going to introduce new metrics for managed memory [1], network memory [2] and metaspace [3] in the upcoming release of Flink 1.12.0. This should make it easier to monitor these memory pools.

 

I hope that helps a bit.

Best,
Matthias

 

[1] https://issues.apache.org/jira/browse/FLINK-14406

[2] https://issues.apache.org/jira/browse/FLINK-14422

[3] https://issues.apache.org/jira/browse/FLINK-19617