Flink task heap doesn't get free after a stream job is failed

[HaochengWang]

My stream job about inserting data stream into HBase using a customized sink table connector failed with the log showing of a one of the task manager lost heartbreak, due to ‘Out of Memory Error’, and when I check the status of the task manager's memory, it seems the task heap is still abnormally at a high usage percent(12.5 / 17.3GB), and maybe lots of objects are not cleared.

What’s the reason of this problem?

[rmetzger0]

Hi,

You might indeed have a memory leak in your customized Hbase sink, or some other dependency or code, or the JVM just didn't trigger a garbage collection yet.

I would first manually trigger a garbage collection on the Taskmanager JVM (like this: https://stackoverflow.com/a/21139655/568695)

If the amout of allocated objects on the heap is really not going down, you can follow these steps to debug classloader leaks:

https://cwiki.apache.org/confluence/display/FLINK/Debugging+ClassLoader+leaks

I hope this helps!

On Fri, Apr 2, 2021 at 10:56 AM 王 浩成 <[hidden email]> wrote:

My stream job about inserting data stream into HBase using a customized sink table connector failed with the log showing of a one of the task manager lost heartbreak, due to ‘Out of Memory Error’, and when I check the status of the task manager's memory, it seems the task heap is still abnormally at a high usage percent(12.5 / 17.3GB), and maybe lots of objects are not cleared.

What’s the reason of this problem?