Re: question on Flink memory management

Posted by Dawid Wysakowicz-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/question-on-Flink-memory-management-tp25160p25161.html

Hi Chang,

The link you've posted describes only the Batch execution. It does not describe the streaming (which I think is what you use, as only there you have access to flink's state). It is also quite old and therefore I am not 100% sure how up-to-date it is.

Best,

Dawid

On 19/12/2018 12:11, Chang Liu wrote:

Dear All,

I am trying to figure how Flink is managing the memory. I found this link:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525

Question 1 - If, in my code, I defined two objects, one is native Java HashMap, one is the Flink MapState, which part of the JVM heap, as mentioned in above figure, each of them is put?

Are both Java HashMap and Flink MapState are put in Free or Memory Manager?

Question 2 - As it is stated in the doc, TaskManager allocates the memory on start-up time (lets say 1024MB). Does that mean, if I check the utilization of the memory on the machine (i.e., via top), these allocated memory is in used? It does not matter whether I have jobs running on the TaskManager?

Lets say, if I start or stop a simple job, the memory usage will the same (if there is not off-heap memory used or there is no OOM)?

Many Thanks.

Best regards/祝好,

Chang Liu 刘畅