Measure the memory consumption of a job at runtime

[Chao Wang]

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao

[kedar mhaswade]

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI). 

(Is there an issue/feature request for this?)

Is that what you are looking for?

On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <[hidden email]> wrote:

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao

[Chao Wang]

Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink and the underlying JVM manage memory..

 

Chao

 

From: kedar mhaswade <[hidden email]>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI). 

(Is there an issue/feature request for this?)

 

 

Is that what you are looking for?

 

 

On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <[hidden email]> wrote:

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao

 

[kedar mhaswade]

I was also able to attach the JMX (jconsole/VisualVM) console to any container (task manager) or job manager with these options in your flink-conf.yaml:

env.java.opts: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=50001 -Dcom.sun.management.jmxremote.rmi.port=50001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=false

That way it shows me how the memory is being allocated and collected. Hope that helps:

On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao <[hidden email]> wrote:

Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink and the underlying JVM manage memory..

 

Chao

 

From: kedar mhaswade <[hidden email]>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI). 

(Is there an issue/feature request for this?)

 

 

Is that what you are looking for?

 

 

On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <[hidden email]> wrote:

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao

 

[Chao Wang]

That helps a lot! Thanks, Kedar!

-Chao

 

From: kedar mhaswade <[hidden email]>
Date: Thursday, February 8, 2018 at 6:37 PM
To: "Wang, Chao" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

I was also able to attach the JMX (jconsole/VisualVM) console to any container (task manager) or job manager with these options in your flink-conf.yaml:

env.java.opts: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=50001 -Dcom.sun.management.jmxremote.rmi.port=50001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=false

 

That way it shows me how the memory is being allocated and collected. Hope that helps:

 

 

On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao <[hidden email]> wrote:

Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink and the underlying JVM manage memory..

 

Chao

 

From: kedar mhaswade <[hidden email]>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI). 

(Is there an issue/feature request for this?)

 

 

Is that what you are looking for?

 

 

On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <[hidden email]> wrote:

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao

 

 

[Kien Truong]

Hi,

Just want to share a tip, for job with rocksdb backend, you can use LD_LIBRARY_PATH to load jemalloc

containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"

Then use jemalloc heap profiler to track native memory usage by rocksdb.

https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling

In my experience, jemalloc helps a lot at reducing memory fragmentation when using rocksdb state backend.

Best regards,

Kien

On 2/9/2018 11:58 PM, Wang, Chao wrote:

That helps a lot! Thanks, Kedar!

-Chao

 

From: kedar mhaswade [hidden email]
Date: Thursday, February 8, 2018 at 6:37 PM
To: "Wang, Chao" [hidden email]
Cc: [hidden email] [hidden email]
Subject: Re: Measure the memory consumption of a job at runtime

 

I was also able to attach the JMX (jconsole/VisualVM) console to any container (task manager) or job manager with these options in your flink-conf.yaml:

env.java.opts: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=50001 -Dcom.sun.management.jmxremote.rmi.port=50001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=false

 

That way it shows me how the memory is being allocated and collected. Hope that helps:

 

 

On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao <[hidden email]> wrote:

Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink and the underlying JVM manage memory..

 

Chao

 

From: kedar mhaswade <[hidden email]>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI). 

(Is there an issue/feature request for this?)

 

 

Is that what you are looking for?

 

 

On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <[hidden email]> wrote:

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao

 

 

[Vishal Santoshi]

This is interesting. Could you elaborate on what you observations have been  and may be flink folks can comment on this approach. RocksDB has been a black box for us and I would assume for more than us, for a long time.

On Tue, Feb 13, 2018 at 5:05 AM, Kien Truong <[hidden email]> wrote:

Hi,

Just want to share a tip, for job with rocksdb backend, you can use LD_LIBRARY_PATH to load jemalloc

containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"

Then use jemalloc heap profiler to track native memory usage by rocksdb.

https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling

In my experience, jemalloc helps a lot at reducing memory fragmentation when using rocksdb state backend.

Best regards,

Kien

On 2/9/2018 11:58 PM, Wang, Chao wrote:

That helps a lot! Thanks, Kedar!

-Chao

 

From: kedar mhaswade [hidden email]
Date: Thursday, February 8, 2018 at 6:37 PM
To: "Wang, Chao" [hidden email]
Cc: [hidden email] [hidden email]
Subject: Re: Measure the memory consumption of a job at runtime

 

I was also able to attach the JMX (jconsole/VisualVM) console to any container (task manager) or job manager with these options in your flink-conf.yaml:

env.java.opts: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=50001 -Dcom.sun.management.jmxremote.rmi.port=50001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=false

 

That way it shows me how the memory is being allocated and collected. Hope that helps:

 

 

On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao <[hidden email]> wrote:

Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink and the underlying JVM manage memory..

 

Chao

 

From: kedar mhaswade <[hidden email]>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI). 

(Is there an issue/feature request for this?)

 

 

Is that what you are looking for?

 

 

On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <[hidden email]> wrote:

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao

 

 

[Chao Wang]

Thanks for the tip, Kien. Alternatively, I measured memory footprint by querying /proc/[pid]/status and by pmap.

 

Best regards,

Chao

 

From: Kien Truong <[hidden email]>
Date: Tuesday, February 13, 2018 at 4:06 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

Hi,

Just want to share a tip, for job with rocksdb backend, you can use LD_LIBRARY_PATH to load jemalloc

containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"

 

Then use jemalloc heap profiler to track native memory usage by rocksdb.

https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling

 

In my experience, jemalloc helps a lot at reducing memory fragmentation when using rocksdb state backend.

 

Best regards,

Kien

 

On 2/9/2018 11:58 PM, Wang, Chao wrote:

That helps a lot! Thanks, Kedar!

-Chao

 

From: kedar mhaswade [hidden email]
Date: Thursday, February 8, 2018 at 6:37 PM
To: "Wang, Chao" [hidden email]
Cc: [hidden email] [hidden email]
Subject: Re: Measure the memory consumption of a job at runtime

 

I was also able to attach the JMX (jconsole/VisualVM) console to any container (task manager) or job manager with these options in your flink-conf.yaml:

env.java.opts: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=50001 -Dcom.sun.management.jmxremote.rmi.port=50001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=false

 

That way it shows me how the memory is being allocated and collected. Hope that helps:

 

 

On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao <[hidden email]> wrote:

Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink and the underlying JVM manage memory..

 

Chao

 

From: kedar mhaswade <[hidden email]>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Measure the memory consumption of a job at runtime

 

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The one capability that is missing is to take the thread dump (ala Spark UI). 

(Is there an issue/feature request for this?)

 

 

Is that what you are looking for?

 

 

On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <[hidden email]> wrote:

Hi,

 

I would like to measure the memory consumption of a job at runtime. I came across some discussion (here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that it’s not possible two years ago. Is it possible in the current status, and if yes, how to do it?

 

Thank you,

 

Chao