Re: JVM Non Heap Memory

Posted by Daniel Santos on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/JVM-Non-Heap-Memory-tp10373p10376.html

Hello,

Nope I am using Hadoop HDFS, as state backend, Kafka, as source, and a HttpClient as a Sink, also Kafka as Sink.
So it's possible that the state backend is the culprit?

Curious thing is even when no jobs are running streaming or otherwise, the JVM Non-HEAP stays the same.
Which I find it odd.

Another curious thing is that it's proportional to an increase of JVM thread's number.
Whenever there are more JVM threads running there is also more JVM Non-HEAP being used, which makes sense.
But threads stick around never decreasing, too, likewise JVM Non-HEAP memory.

These observations described are based on what flink's metrics are being sent and recorded to our graphite's system.

Best Regards,

Daniel Santos

On 11/29/2016 04:04 PM, Cliff Resnick wrote:

Are you using the RocksDB backend in native mode? If so then the off-heap memory may be there.

On Tue, Nov 29, 2016 at 9:54 AM, <[hidden email]> wrote:

i have the same problem,but i put the flink job into yarn.
but i put the job into yarn on the computer 22,and the job can success run,and the jobmanager is 79 and taskmanager is 69,they three different compu345ter,
however,on computer 22,the pid=3463,which is the job that put into yarn,is have 2.3g memory,15% of total,
the commend is : ./flink run -m yarn-cluster -yn 1 -ys 1 -yjm 1024 -ytm 1024 ....
why in conputer 22,has occupy so much momory?the job is running computer 79 and computer 69.
What would be the possible causes of such behavior ?
Best Regards,

----- 原始邮件 -----
发件人：Daniel Santos <[hidden email]>
收件人：[hidden email]
主题：JVM Non Heap Memory
日期：2016年11月29日 22点26分

Hello,
Is it common to have high usage of Non-Heap in JVM ?
I am running flink in stand-alone cluster and in docker, with each
docker bieng capped at 6G of memory.
I have been struggling to keep memory usage in check.
The non-heap increases to no end. It start with just 100MB of usage and
after a day it reaches to 1,3GB.
Then evetually reaches to 2GB and then eventually the docker is killed
because it has reached the memory limit.
My configuration for each flink task manager is the following :
----------- flink-conf.yaml --------------
taskmanager.heap.mb: 3072
taskmanager.numberOfTaskSlots: 8
taskmanager.memory.preallocate: false
taskmanager.network.numberOfBuffers: 12500
taskmanager.memory.off-heap: false
---------------------------------------------
What would be the possible causes of such behavior ?
Best Regards,
Daniel Santos