period batch job lead to OutOfMemoryError: Metaspace problem

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

period batch job lead to OutOfMemoryError: Metaspace problem

太平洋
batch job:
read data from s3 by sql,then by some operators and write data to clickhouse and kafka.
after some times, task-manager quit with OutOfMemoryError: Metaspace.

env:
flink version:1.12.2
task-manager slot count: 5
deployment: standalone kubernetes session 模式
dependencies:

    <dependency>

      <groupId>org.apache.flink</groupId>

      <artifactId>flink-connector-kafka_2.11</artifactId>

      <version>${flink.version}</version>

    </dependency>

    <dependency>

      <groupId>com.google.code.gson</groupId>

      <artifactId>gson</artifactId>

      <version>2.8.5</version>

    </dependency>

    <dependency>

      <groupId>org.apache.flink</groupId>

      <artifactId>flink-connector-jdbc_2.11</artifactId>

      <version>${flink.version}</version>   

    </dependency>

    <dependency>

      <groupId>ru.yandex.clickhouse</groupId>

      <artifactId>clickhouse-jdbc</artifactId>

      <version>0.3.0</version>

    </dependency>

    <dependency>

      <groupId>org.apache.flink</groupId>

        <artifactId>flink-parquet_2.11</artifactId>

        <version>${flink.version}</version>

    </dependency>

    <dependency>

         <groupId>org.apache.flink</groupId>

         <artifactId>flink-json</artifactId>

         <version>${flink.version}</version>

    </dependency>


heap dump1:

Leak Suspects

System Overview

 Leaks 

 Overview

  Problem Suspect 1

21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 29,656,880 (41.16%) bytes. 

Biggest instances:

  • org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8 - 1,474,760 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820 - 1,474,168 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10 - 1,474,160 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0 - 1,474,160 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8 - 1,474,160 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108 - 1,474,128 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0 - 1,474,120 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778 - 1,474,112 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8 - 1,474,064 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38 - 1,474,064 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638 - 1,474,064 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418 - 1,474,056 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60 - 1,474,056 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38 - 1,474,056 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8 - 1,474,032 (2.05%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8 - 1,461,944 (2.03%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98 - 1,460,752 (2.03%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0 - 1,460,744 (2.03%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8 - 1,445,232 (2.01%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00 - 1,445,232 (2.01%) bytes. 


Keywords
org.apache.flink.util.ChildFirstClassLoader
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0

  Problem Suspect 2

34,407 instances of "org.apache.flink.core.memory.HybridMemorySegment", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 7,707,168 (10.70%) bytes. 

Keywords
org.apache.flink.core.memory.HybridMemorySegment
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0



heap dump2:

Leak Suspects

System Overview

 Leaks 

 Overview

  Problem Suspect 1

21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 26,061,408 (30.68%) bytes. 

Biggest instances:

  • org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930 - 1,474,224 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8 - 1,474,224 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0 - 1,474,168 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118 - 1,474,168 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0 - 1,474,168 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28 - 1,474,160 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0 - 1,474,160 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8 - 1,474,136 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0 - 1,474,064 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0 - 1,474,064 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860 - 1,474,064 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768 - 1,474,064 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878 - 1,474,056 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640 - 1,474,056 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80 - 1,474,040 (1.74%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868 - 1,469,720 (1.73%) bytes. 
  • org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98 - 1,460,808 (1.72%) bytes. 


Keywords
org.apache.flink.util.ChildFirstClassLoader
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0

  Problem Suspect 2

4 instances of "org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 11,644,200 (13.71%) bytes. 

Biggest instances:

  • org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73e2d0cb0 - 4,364,536 (5.14%) bytes. 
  • org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73d62fb88 - 3,643,576 (4.29%) bytes. 
  • org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73dae0270 - 3,635,952 (4.28%) bytes. 


Keywords
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask


Reply | Threaded
Open this post in threaded view
|

Re: period batch job lead to OutOfMemoryError: Metaspace problem

Yangze Guo
I think you can try to increase the JVM metaspace option for
TaskManagers through taskmanager.memory.jvm-metaspace.size. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-metaspace

Best,
Yangze Guo

Best,
Yangze Guo


On Tue, Apr 6, 2021 at 4:22 PM 太平洋 <[hidden email]> wrote:

>
> batch job:
> read data from s3 by sql,then by some operators and write data to clickhouse and kafka.
> after some times, task-manager quit with OutOfMemoryError: Metaspace.
>
> env:
> flink version:1.12.2
> task-manager slot count: 5
> deployment: standalone kubernetes session 模式
> dependencies:
>
>     <dependency>
>
>       <groupId>org.apache.flink</groupId>
>
>       <artifactId>flink-connector-kafka_2.11</artifactId>
>
>       <version>${flink.version}</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>com.google.code.gson</groupId>
>
>       <artifactId>gson</artifactId>
>
>       <version>2.8.5</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>org.apache.flink</groupId>
>
>       <artifactId>flink-connector-jdbc_2.11</artifactId>
>
>       <version>${flink.version}</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>ru.yandex.clickhouse</groupId>
>
>       <artifactId>clickhouse-jdbc</artifactId>
>
>       <version>0.3.0</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>org.apache.flink</groupId>
>
>         <artifactId>flink-parquet_2.11</artifactId>
>
>         <version>${flink.version}</version>
>
>     </dependency>
>
>     <dependency>
>
>          <groupId>org.apache.flink</groupId>
>
>          <artifactId>flink-json</artifactId>
>
>          <version>${flink.version}</version>
>
>     </dependency>
>
>
> heap dump1:
>
> Leak Suspects
>
> System Overview
>
>  Leaks
>
>  Overview
>
>
>   Problem Suspect 1
>
> 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 29,656,880 (41.16%) bytes.
>
> Biggest instances:
>
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8 - 1,474,760 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820 - 1,474,168 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10 - 1,474,160 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0 - 1,474,160 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8 - 1,474,160 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108 - 1,474,128 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0 - 1,474,120 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778 - 1,474,112 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8 - 1,474,064 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38 - 1,474,064 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638 - 1,474,064 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418 - 1,474,056 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60 - 1,474,056 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38 - 1,474,056 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8 - 1,474,032 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8 - 1,461,944 (2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98 - 1,460,752 (2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0 - 1,460,744 (2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8 - 1,445,232 (2.01%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00 - 1,445,232 (2.01%) bytes.
>
>
>
> Keywords
> org.apache.flink.util.ChildFirstClassLoader
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> Details »
>
>   Problem Suspect 2
>
> 34,407 instances of "org.apache.flink.core.memory.HybridMemorySegment", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 7,707,168 (10.70%) bytes.
>
> Keywords
> org.apache.flink.core.memory.HybridMemorySegment
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
>
> Details »
>
>
>
> heap dump2:
>
> Leak Suspects
>
> System Overview
>
>  Leaks
>
>  Overview
>
>   Problem Suspect 1
>
> 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 26,061,408 (30.68%) bytes.
>
> Biggest instances:
>
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930 - 1,474,224 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8 - 1,474,224 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0 - 1,474,168 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118 - 1,474,168 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0 - 1,474,168 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28 - 1,474,160 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0 - 1,474,160 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8 - 1,474,136 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878 - 1,474,056 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640 - 1,474,056 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80 - 1,474,040 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868 - 1,469,720 (1.73%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98 - 1,460,808 (1.72%) bytes.
>
>
>
> Keywords
> org.apache.flink.util.ChildFirstClassLoader
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> Details »
>
>   Problem Suspect 2
>
> 4 instances of "org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 11,644,200 (13.71%) bytes.
>
> Biggest instances:
>
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73e2d0cb0 - 4,364,536 (5.14%) bytes.
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73d62fb88 - 3,643,576 (4.29%) bytes.
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73dae0270 - 3,635,952 (4.28%) bytes.
>
>
>
> Keywords
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask
> Details »
>
>
Reply | Threaded
Open this post in threaded view
|

回复: period batch job lead to OutOfMemoryError: Metaspace problem

太平洋
I have tried this method, but the problem still exist.
by heap dump analysis, is 21 instances of "org.apache.flink.util.ChildFirstClassLoader" normal?


------------------ 原始邮件 ------------------
发件人: "Yangze Guo" <[hidden email]>;
发送时间: 2021年4月6日(星期二) 下午4:32
收件人: "太平洋"<[hidden email]>;
抄送: "user"<[hidden email]>;
主题: Re: period batch job lead to OutOfMemoryError: Metaspace problem

I think you can try to increase the JVM metaspace option for
TaskManagers through taskmanager.memory.jvm-metaspace.size. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-metaspace

Best,
Yangze Guo

Best,
Yangze Guo


On Tue, Apr 6, 2021 at 4:22 PM 太平洋 <[hidden email]> wrote:

>
> batch job:
> read data from s3 by sql,then by some operators and write data to clickhouse and kafka.
> after some times, task-manager quit with OutOfMemoryError: Metaspace.
>
> env:
> flink version:1.12.2
> task-manager slot count: 5
> deployment: standalone kubernetes session 模式
> dependencies:
>
>     <dependency>
>
>       <groupId>org.apache.flink</groupId>
>
>       <artifactId>flink-connector-kafka_2.11</artifactId>
>
>       <version>${flink.version}</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>com.google.code.gson</groupId>
>
>       <artifactId>gson</artifactId>
>
>       <version>2.8.5</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>org.apache.flink</groupId>
>
>       <artifactId>flink-connector-jdbc_2.11</artifactId>
>
>       <version>${flink.version}</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>ru.yandex.clickhouse</groupId>
>
>       <artifactId>clickhouse-jdbc</artifactId>
>
>       <version>0.3.0</version>
>
>     </dependency>
>
>     <dependency>
>
>       <groupId>org.apache.flink</groupId>
>
>         <artifactId>flink-parquet_2.11</artifactId>
>
>         <version>${flink.version}</version>
>
>     </dependency>
>
>     <dependency>
>
>          <groupId>org.apache.flink</groupId>
>
>          <artifactId>flink-json</artifactId>
>
>          <version>${flink.version}</version>
>
>     </dependency>
>
>
> heap dump1:
>
> Leak Suspects
>
> System Overview
>
>  Leaks
>
>  Overview
>
>
>   Problem Suspect 1
>
> 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 29,656,880 (41.16%) bytes.
>
> Biggest instances:
>
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8 - 1,474,760 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820 - 1,474,168 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10 - 1,474,160 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0 - 1,474,160 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8 - 1,474,160 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108 - 1,474,128 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0 - 1,474,120 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778 - 1,474,112 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8 - 1,474,064 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38 - 1,474,064 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638 - 1,474,064 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418 - 1,474,056 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60 - 1,474,056 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38 - 1,474,056 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8 - 1,474,032 (2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8 - 1,461,944 (2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98 - 1,460,752 (2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0 - 1,460,744 (2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8 - 1,445,232 (2.01%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00 - 1,445,232 (2.01%) bytes.
>
>
>
> Keywords
> org.apache.flink.util.ChildFirstClassLoader
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> Details »
>
>   Problem Suspect 2
>
> 34,407 instances of "org.apache.flink.core.memory.HybridMemorySegment", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 7,707,168 (10.70%) bytes.
>
> Keywords
> org.apache.flink.core.memory.HybridMemorySegment
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
>
> Details »
>
>
>
> heap dump2:
>
> Leak Suspects
>
> System Overview
>
>  Leaks
>
>  Overview
>
>   Problem Suspect 1
>
> 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 26,061,408 (30.68%) bytes.
>
> Biggest instances:
>
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930 - 1,474,224 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8 - 1,474,224 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0 - 1,474,168 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118 - 1,474,168 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0 - 1,474,168 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28 - 1,474,160 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0 - 1,474,160 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8 - 1,474,136 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768 - 1,474,064 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878 - 1,474,056 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640 - 1,474,056 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80 - 1,474,040 (1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868 - 1,469,720 (1.73%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98 - 1,460,808 (1.72%) bytes.
>
>
>
> Keywords
> org.apache.flink.util.ChildFirstClassLoader
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> Details »
>
>   Problem Suspect 2
>
> 4 instances of "org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 11,644,200 (13.71%) bytes.
>
> Biggest instances:
>
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73e2d0cb0 - 4,364,536 (5.14%) bytes.
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73d62fb88 - 3,643,576 (4.29%) bytes.
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73dae0270 - 3,635,952 (4.28%) bytes.
>
>
>
> Keywords
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask
> Details »
>
>
Reply | Threaded
Open this post in threaded view
|

Re: period batch job lead to OutOfMemoryError: Metaspace problem

Yangze Guo
> I have tried this method, but the problem still exist.
How much memory do you configure for it?

> is 21 instances of "org.apache.flink.util.ChildFirstClassLoader" normal
Not quite sure about it. AFAIK, each job will have a classloader.
Multiple tasks of the same job in the same TM will share the same
classloader. The classloader will be removed if there is no more task
running on the TM. Classloader without reference will be finally
cleanup by GC. Could you share JM and TM logs for further analysis?
I'll also involve @Guowei Ma in this thread.


Best,
Yangze Guo

On Tue, Apr 6, 2021 at 6:05 PM 太平洋 <[hidden email]> wrote:

>
> I have tried this method, but the problem still exist.
> by heap dump analysis, is 21 instances of "org.apache.flink.util.ChildFirstClassLoader" normal?
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Yangze Guo" <[hidden email]>;
> 发送时间: 2021年4月6日(星期二) 下午4:32
> 收件人: "太平洋"<[hidden email]>;
> 抄送: "user"<[hidden email]>;
> 主题: Re: period batch job lead to OutOfMemoryError: Metaspace problem
>
> I think you can try to increase the JVM metaspace option for
> TaskManagers through taskmanager.memory.jvm-metaspace.size. [1]
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-metaspace
>
> Best,
> Yangze Guo
>
> Best,
> Yangze Guo
>
>
> On Tue, Apr 6, 2021 at 4:22 PM 太平洋 <[hidden email]> wrote:
> >
> > batch job:
> > read data from s3 by sql,then by some operators and write data to clickhouse and kafka.
> > after some times, task-manager quit with OutOfMemoryError: Metaspace.
> >
> > env:
> > flink version:1.12.2
> > task-manager slot count: 5
> > deployment: standalone kubernetes session 模式
> > dependencies:
> >
> >     <dependency>
> >
> >       <groupId>org.apache.flink</groupId>
> >
> >       <artifactId>flink-connector-kafka_2.11</artifactId>
> >
> >       <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>com.google.code.gson</groupId>
> >
> >       <artifactId>gson</artifactId>
> >
> >       <version>2.8.5</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>org.apache.flink</groupId>
> >
> >       <artifactId>flink-connector-jdbc_2.11</artifactId>
> >
> >       <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>ru.yandex.clickhouse</groupId>
> >
> >       <artifactId>clickhouse-jdbc</artifactId>
> >
> >       <version>0.3.0</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>org.apache.flink</groupId>
> >
> >         <artifactId>flink-parquet_2.11</artifactId>
> >
> >         <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >          <groupId>org.apache.flink</groupId>
> >
> >          <artifactId>flink-json</artifactId>
> >
> >          <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >
> > heap dump1:
> >
> > Leak Suspects
> >
> > System Overview
> >
> >  Leaks
> >
> >  Overview
> >
> >
> >   Problem Suspect 1
> >
> > 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 29,656,880 (41.16%) bytes.
> >
> > Biggest instances:
> >
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8 - 1,474,760 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820 - 1,474,168 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10 - 1,474,160 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0 - 1,474,160 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8 - 1,474,160 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108 - 1,474,128 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0 - 1,474,120 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778 - 1,474,112 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8 - 1,474,064 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38 - 1,474,064 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638 - 1,474,064 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418 - 1,474,056 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60 - 1,474,056 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38 - 1,474,056 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8 - 1,474,032 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8 - 1,461,944 (2.03%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98 - 1,460,752 (2.03%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0 - 1,460,744 (2.03%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8 - 1,445,232 (2.01%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00 - 1,445,232 (2.01%) bytes.
> >
> >
> >
> > Keywords
> > org.apache.flink.util.ChildFirstClassLoader
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> > Details »
> >
> >   Problem Suspect 2
> >
> > 34,407 instances of "org.apache.flink.core.memory.HybridMemorySegment", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 7,707,168 (10.70%) bytes.
> >
> > Keywords
> > org.apache.flink.core.memory.HybridMemorySegment
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> >
> > Details »
> >
> >
> >
> > heap dump2:
> >
> > Leak Suspects
> >
> > System Overview
> >
> >  Leaks
> >
> >  Overview
> >
> >   Problem Suspect 1
> >
> > 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 26,061,408 (30.68%) bytes.
> >
> > Biggest instances:
> >
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930 - 1,474,224 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8 - 1,474,224 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0 - 1,474,168 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118 - 1,474,168 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0 - 1,474,168 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28 - 1,474,160 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0 - 1,474,160 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8 - 1,474,136 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878 - 1,474,056 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640 - 1,474,056 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80 - 1,474,040 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868 - 1,469,720 (1.73%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98 - 1,460,808 (1.72%) bytes.
> >
> >
> >
> > Keywords
> > org.apache.flink.util.ChildFirstClassLoader
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> > Details »
> >
> >   Problem Suspect 2
> >
> > 4 instances of "org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 11,644,200 (13.71%) bytes.
> >
> > Biggest instances:
> >
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73e2d0cb0 - 4,364,536 (5.14%) bytes.
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73d62fb88 - 3,643,576 (4.29%) bytes.
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73dae0270 - 3,635,952 (4.28%) bytes.
> >
> >
> >
> > Keywords
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask
> > Details »
> >
> >
Reply | Threaded
Open this post in threaded view
|

回复: period batch job lead to OutOfMemoryError: Metaspace problem

太平洋
I have configured to 512M, but problem still exist. Now the memory size is still 256M.




------------------ 原始邮件 ------------------
发件人: "Yangze Guo" <[hidden email]>;
发送时间: 2021年4月6日(星期二) 晚上6:35
收件人: "太平洋"<[hidden email]>;
抄送: "user"<[hidden email]>;"guowei.mgw"<[hidden email]>;
主题: Re: period batch job lead to OutOfMemoryError: Metaspace problem

> I have tried this method, but the problem still exist.
How much memory do you configure for it?

> is 21 instances of "org.apache.flink.util.ChildFirstClassLoader" normal
Not quite sure about it. AFAIK, each job will have a classloader.
Multiple tasks of the same job in the same TM will share the same
classloader. The classloader will be removed if there is no more task
running on the TM. Classloader without reference will be finally
cleanup by GC. Could you share JM and TM logs for further analysis?
I'll also involve @Guowei Ma in this thread.


Best,
Yangze Guo

On Tue, Apr 6, 2021 at 6:05 PM 太平洋 <[hidden email]> wrote:

>
> I have tried this method, but the problem still exist.
> by heap dump analysis, is 21 instances of "org.apache.flink.util.ChildFirstClassLoader" normal?
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Yangze Guo" <[hidden email]>;
> 发送时间: 2021年4月6日(星期二) 下午4:32
> 收件人: "太平洋"<[hidden email]>;
> 抄送: "user"<[hidden email]>;
> 主题: Re: period batch job lead to OutOfMemoryError: Metaspace problem
>
> I think you can try to increase the JVM metaspace option for
> TaskManagers through taskmanager.memory.jvm-metaspace.size. [1]
>
> [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-metaspace
>
> Best,
> Yangze Guo
>
> Best,
> Yangze Guo
>
>
> On Tue, Apr 6, 2021 at 4:22 PM 太平洋 <[hidden email]> wrote:
> >
> > batch job:
> > read data from s3 by sql,then by some operators and write data to clickhouse and kafka.
> > after some times, task-manager quit with OutOfMemoryError: Metaspace.
> >
> > env:
> > flink version:1.12.2
> > task-manager slot count: 5
> > deployment: standalone kubernetes session 模式
> > dependencies:
> >
> >     <dependency>
> >
> >       <groupId>org.apache.flink</groupId>
> >
> >       <artifactId>flink-connector-kafka_2.11</artifactId>
> >
> >       <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>com.google.code.gson</groupId>
> >
> >       <artifactId>gson</artifactId>
> >
> >       <version>2.8.5</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>org.apache.flink</groupId>
> >
> >       <artifactId>flink-connector-jdbc_2.11</artifactId>
> >
> >       <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>ru.yandex.clickhouse</groupId>
> >
> >       <artifactId>clickhouse-jdbc</artifactId>
> >
> >       <version>0.3.0</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >       <groupId>org.apache.flink</groupId>
> >
> >         <artifactId>flink-parquet_2.11</artifactId>
> >
> >         <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >     <dependency>
> >
> >          <groupId>org.apache.flink</groupId>
> >
> >          <artifactId>flink-json</artifactId>
> >
> >          <version>${flink.version}</version>
> >
> >     </dependency>
> >
> >
> > heap dump1:
> >
> > Leak Suspects
> >
> > System Overview
> >
> >  Leaks
> >
> >  Overview
> >
> >
> >   Problem Suspect 1
> >
> > 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 29,656,880 (41.16%) bytes.
> >
> > Biggest instances:
> >
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8 - 1,474,760 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820 - 1,474,168 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10 - 1,474,160 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0 - 1,474,160 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8 - 1,474,160 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108 - 1,474,128 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0 - 1,474,120 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778 - 1,474,112 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8 - 1,474,064 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38 - 1,474,064 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638 - 1,474,064 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418 - 1,474,056 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60 - 1,474,056 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38 - 1,474,056 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8 - 1,474,032 (2.05%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8 - 1,461,944 (2.03%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98 - 1,460,752 (2.03%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0 - 1,460,744 (2.03%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8 - 1,445,232 (2.01%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00 - 1,445,232 (2.01%) bytes.
> >
> >
> >
> > Keywords
> > org.apache.flink.util.ChildFirstClassLoader
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> > Details »
> >
> >   Problem Suspect 2
> >
> > 34,407 instances of "org.apache.flink.core.memory.HybridMemorySegment", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 7,707,168 (10.70%) bytes.
> >
> > Keywords
> > org.apache.flink.core.memory.HybridMemorySegment
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> >
> > Details »
> >
> >
> >
> > heap dump2:
> >
> > Leak Suspects
> >
> > System Overview
> >
> >  Leaks
> >
> >  Overview
> >
> >   Problem Suspect 1
> >
> > 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 26,061,408 (30.68%) bytes.
> >
> > Biggest instances:
> >
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930 - 1,474,224 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8 - 1,474,224 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0 - 1,474,168 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118 - 1,474,168 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0 - 1,474,168 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28 - 1,474,160 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0 - 1,474,160 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8 - 1,474,136 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768 - 1,474,064 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878 - 1,474,056 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640 - 1,474,056 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80 - 1,474,040 (1.74%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868 - 1,469,720 (1.73%) bytes.
> > org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98 - 1,460,808 (1.72%) bytes.
> >
> >
> >
> > Keywords
> > org.apache.flink.util.ChildFirstClassLoader
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> > Details »
> >
> >   Problem Suspect 2
> >
> > 4 instances of "org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 11,644,200 (13.71%) bytes.
> >
> > Biggest instances:
> >
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73e2d0cb0 - 4,364,536 (5.14%) bytes.
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73d62fb88 - 3,643,576 (4.29%) bytes.
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73dae0270 - 3,635,952 (4.28%) bytes.
> >
> >
> >
> > Keywords
> > sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask
> > Details »
> >
> >

tm.log (2M) Download Attachment
jm.log (7M) Download Attachment