question on Flink memory management

[Chang Liu]

Dear All,

I am trying to figure how Flink is managing the memory. I found this link:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525

Question 1 - If, in my code, I defined two objects, one is native Java HashMap, one is the Flink MapState, which part of the JVM heap, as mentioned in above figure, each of them is put?

Are both Java HashMap and Flink MapState are put in Free or Memory Manager?

Question 2 - As it is stated in the doc, TaskManager allocates the memory on start-up time (lets say 1024MB). Does that mean, if I check the utilization of the memory on the machine (i.e., via top), these allocated memory is in used? It does not matter whether I have jobs running on the TaskManager?

Lets say, if I start or stop a simple job, the memory usage will the same (if there is not off-heap memory used or there is no OOM)?

Many Thanks.

Best regards/祝好,

Chang Liu 刘畅

[Dawid Wysakowicz-2]

Hi Chang,

The link you've posted describes only the Batch execution. It does not describe the streaming (which I think is what you use, as only there you have access to flink's state). It is also quite old and therefore I am not 100% sure how up-to-date it is.

Best,

Dawid

On 19/12/2018 12:11, Chang Liu wrote:

Dear All,

I am trying to figure how Flink is managing the memory. I found this link:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525

Question 1 - If, in my code, I defined two objects, one is native Java HashMap, one is the Flink MapState, which part of the JVM heap, as mentioned in above figure, each of them is put?

Are both Java HashMap and Flink MapState are put in Free or Memory Manager?

Question 2 - As it is stated in the doc, TaskManager allocates the memory on start-up time (lets say 1024MB). Does that mean, if I check the utilization of the memory on the machine (i.e., via top), these allocated memory is in used? It does not matter whether I have jobs running on the TaskManager?

Lets say, if I start or stop a simple job, the memory usage will the same (if there is not off-heap memory used or there is no OOM)?

Many Thanks.

Best regards/祝好,

Chang Liu 刘畅

[Chang Liu]

Hi Dawid,

Thanks for your reply.

Then, we can just ignore the link and just focus on the questions.

Best regards/祝好,

Chang Liu 刘畅

On 19 Dec 2018, at 12:18, Dawid Wysakowicz <[hidden email]> wrote:

Hi Chang,

The link you've posted describes only the Batch execution. It does not describe the streaming (which I think is what you use, as only there you have access to flink's state). It is also quite old and therefore I am not 100% sure how up-to-date it is.

Best,

Dawid

On 19/12/2018 12:11, Chang Liu wrote:

Dear All,

I am trying to figure how Flink is managing the memory. I found this link:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525

<HeapDivision.png>

Question 1 - If, in my code, I defined two objects, one is native Java HashMap, one is the Flink MapState, which part of the JVM heap, as mentioned in above figure, each of them is put?

Are both Java HashMap and Flink MapState are put in Free or Memory Manager?

Question 2 - As it is stated in the doc, TaskManager allocates the memory on start-up time (lets say 1024MB). Does that mean, if I check the utilization of the memory on the machine (i.e., via top), these allocated memory is in used? It does not matter whether I have jobs running on the TaskManager?

Lets say, if I start or stop a simple job, the memory usage will the same (if there is not off-heap memory used or there is no OOM)?

Many Thanks.

Best regards/祝好,

Chang Liu 刘畅

[Chang Liu]

I found this answer https://stackoverflow.com/a/50816868/4802797 from Till.

And he mentioned there that "...streaming jobs currently don't use managed memory…".

So that means, the memory used by flink streaming is really based on the jobs running there. If I cancel the job, the memory utilization should be dropped, right?

Best regards/祝好,

Chang Liu 刘畅

On 19 Dec 2018, at 13:50, Chang Liu <[hidden email]> wrote:

Hi Dawid,

Thanks for your reply.

Then, we can just ignore the link and just focus on the questions.

Best regards/祝好,

Chang Liu 刘畅

On 19 Dec 2018, at 12:18, Dawid Wysakowicz <[hidden email]> wrote:

Hi Chang,

The link you've posted describes only the Batch execution. It does not describe the streaming (which I think is what you use, as only there you have access to flink's state). It is also quite old and therefore I am not 100% sure how up-to-date it is.

Best,

Dawid

On 19/12/2018 12:11, Chang Liu wrote:

Dear All,

I am trying to figure how Flink is managing the memory. I found this link:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525

<HeapDivision.png>

Question 1 - If, in my code, I defined two objects, one is native Java HashMap, one is the Flink MapState, which part of the JVM heap, as mentioned in above figure, each of them is put?

Are both Java HashMap and Flink MapState are put in Free or Memory Manager?

Question 2 - As it is stated in the doc, TaskManager allocates the memory on start-up time (lets say 1024MB). Does that mean, if I check the utilization of the memory on the machine (i.e., via top), these allocated memory is in used? It does not matter whether I have jobs running on the TaskManager?

Lets say, if I start or stop a simple job, the memory usage will the same (if there is not off-heap memory used or there is no OOM)?

Many Thanks.

Best regards/祝好,

Chang Liu 刘畅