Random Task executor shutdown (java.lang.OutOfMemoryError: Metaspace)

[Flavio Pompermaier]

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

[Jan Lukavský]

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

[Flavio Pompermaier]

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

[Jan Lukavský]

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

[Flavio Pompermaier]

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

[Jan Lukavský]

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

[Kye Bae]

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

[Flavio Pompermaier]

Thank you Kye for your insights...in my mind, if the job runs without problems one or more times the heap size, and thus the medatadata-size, is big enough and I should not increase it (on the same data of course).

So I'll try to understand who is leaking what..the advice to avoid the dynamic class loading is just a workaround to me..there's something wrong going on and tomorrow I'll try to understand the root cause of the problem using -XX:NativeMemoryTracking=summary as you suggested.

I'll keep you up to date with my findings..

Best,

Flavio

On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <[hidden email]> wrote:

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

[Flavio Pompermaier]

Another big potential candidate is the fact that JDBC libs I use in my job are put into the Flink lib folder instead of putting them into the fat jar..tomorrow I'll try to see if the metaspace is getting cleared correctly after that change.

Unfortunately our jobs were written before the child-first / parent-first classloading refactoring and at that time that was the way to go..but now it can cause this kind of problems if using child-first policy.

On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <[hidden email]> wrote:

Thank you Kye for your insights...in my mind, if the job runs without problems one or more times the heap size, and thus the medatadata-size, is big enough and I should not increase it (on the same data of course).

So I'll try to understand who is leaking what..the advice to avoid the dynamic class loading is just a workaround to me..there's something wrong going on and tomorrow I'll try to understand the root cause of the problem using -XX:NativeMemoryTracking=summary as you suggested.

I'll keep you up to date with my findings..

Best,

Flavio

On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <[hidden email]> wrote:

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

[Kye Bae]

It is possible, but I am not entirely sure about the load order affecting the metaspace usage.

To find out why your taskmanager container is exceeding the metaspace, we would need to know what value the max metaspace size is set to and then find out how much of the metaspace is actually being used and what the classloading stats are (before Yarn kills the container). 

If possible, please share the current metaspace setting and the native memory stats from one of your taskmanager instances (Java process as a Yarn container). I think those run as the yarn user as opposed to the hadoop user (it depends on your environment).

# enable the native memory tracking referenced earlier - this may need to be passed in as the taskmanager JVM option parameter not the jobmanager.

# log onto one of the taskmanager HW instances

sudo su - yarn # or log on as the user that runs Yarn

jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner - this would be a taskmanager

jcmd TM_PID VM.native_memory summary # size information for "Class" - metaspace

jcmd TM_PID VM.classloader_stats # how many classes were loaded by which classloader, sizes, etc.

There is a gentleman who posts blogs on Flink, and he goes into a bit more detail there. This was against Flink 1.9, but the foundational concepts still apply to 1.10. http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/

-K

On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <[hidden email]> wrote:

Another big potential candidate is the fact that JDBC libs I use in my job are put into the Flink lib folder instead of putting them into the fat jar..tomorrow I'll try to see if the metaspace is getting cleared correctly after that change.

Unfortunately our jobs were written before the child-first / parent-first classloading refactoring and at that time that was the way to go..but now it can cause this kind of problems if using child-first policy.

On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <[hidden email]> wrote:

Thank you Kye for your insights...in my mind, if the job runs without problems one or more times the heap size, and thus the medatadata-size, is big enough and I should not increase it (on the same data of course).

So I'll try to understand who is leaking what..the advice to avoid the dynamic class loading is just a workaround to me..there's something wrong going on and tomorrow I'll try to understand the root cause of the problem using -XX:NativeMemoryTracking=summary as you suggested.

I'll keep you up to date with my findings..

Best,

Flavio

On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <[hidden email]> wrote:

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

[Flavio Pompermaier]

Hi Kye, thanks a lot for the support! See my answers inline below

On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <[hidden email]> wrote:

It is possible, but I am not entirely sure about the load order affecting the metaspace usage.

To find out why your taskmanager container is exceeding the metaspace, we would need to know what value the max metaspace size is set to and then find out how much of the metaspace is actually being used and what the classloading stats are (before Yarn kills the container). 

If possible, please share the current metaspace setting and the native memory stats from one of your taskmanager instances (Java process as a Yarn container). I think those run as the yarn user as opposed to the hadoop user (it depends on your environment).

 

Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize is automatically computed and it is set to 256 MB (-XX:MaxMetaspaceSize=268435456)

# enable the native memory tracking referenced earlier - this may need to be passed in as the taskmanager JVM option parameter not the jobmanager.

# log onto one of the taskmanager HW instances

sudo su - yarn # or log on as the user that runs Yarn

jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner - this would be a taskmanager

jcmd TM_PID VM.native_memory summary # size information for "Class" - metaspace

 jcmd 6976 VM.native_memory summary
6976:

Native Memory Tracking:

Total: reserved=1473311KB, committed=954707KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB)
 
-                     Class (reserved=379640KB, committed=142024KB)
                            (classes #23605)
                            (  instance classes #22058, array classes #1547)
                            (malloc=2808KB #48655)
                            (mmap: reserved=376832KB, committed=139216KB)
                            (  Metadata:   )
                            (    reserved=122880KB, committed=121168KB)
                            (    used=117529KB)
                            (    free=3639KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=253952KB, committed=18048KB)
                            (    used=16516KB)
                            (    free=1532KB)
                            (    waste=0KB =0.00%)
 
-                    Thread (reserved=76359KB, committed=6867KB)
                            (thread #74)
                            (stack: reserved=76012KB, committed=6520KB)
                            (malloc=262KB #446)
                            (arena=85KB #146)
 
-                      Code (reserved=250407KB, committed=38911KB)
                            (malloc=2719KB #11989)
                            (mmap: reserved=247688KB, committed=36192KB)
 
-                        GC (reserved=71481KB, committed=71481KB)
                            (malloc=19121KB #17323)
                            (mmap: reserved=52360KB, committed=52360KB)
 
-                  Compiler (reserved=561KB, committed=561KB)
                            (malloc=429KB #846)
                            (arena=133KB #5)
 
-                  Internal (reserved=855KB, committed=855KB)
                            (malloc=823KB #2442)
                            (mmap: reserved=32KB, committed=32KB)
 
-                     Other (reserved=131857KB, committed=131857KB)
                            (malloc=131857KB #4128)
 
-                    Symbol (reserved=15702KB, committed=15702KB)
                            (malloc=14288KB #165850)
                            (arena=1415KB #1)
 
-    Native Memory Tracking (reserved=4072KB, committed=4072KB)
                            (malloc=27KB #353)
                            (tracking overhead=4045KB)
 
-        Shared class space (reserved=17044KB, committed=17044KB)
                            (mmap: reserved=17044KB, committed=17044KB)
 
-               Arena Chunk (reserved=199KB, committed=199KB)
                            (malloc=199KB)
 
-                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #191)
 
-                 Arguments (reserved=20KB, committed=20KB)
                            (malloc=20KB #501)
 
-                    Module (reserved=568KB, committed=568KB)
                            (malloc=568KB #3919)
 
-              Synchronizer (reserved=244KB, committed=244KB)
                            (malloc=244KB #2054)
 
-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

 

jcmd TM_PID VM.classloader_stats # how many classes were loaded by which classloader, sizes, etc.

 jcmd 6976 VM.classloader_stats
6976:
ClassLoader         Parent              CLD*               Classes   ChunkSz   BlockSz  Type
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d8140       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38193d50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00a9f10       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ab290       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38011150       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5d90       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004dec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ae230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402a2f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c5803b430       1      4096      3232  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c072ce0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800b7ee0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc336610       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084aa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc0011ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2f70       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021090       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8085cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d3830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00becf0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812a7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c880       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc018030       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139a90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90020cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9806f6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804c2d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80bcf80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc347ec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01aca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38576590       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380f71f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc40110d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c030       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2440       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c8003c3a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bb00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900cee70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053d30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b5a00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e29b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053250       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60057820       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867f350       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d765c0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc4014c00       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2f90       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae450       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc014ae0       1     10240      7392  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813e880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036ef80  0x00007f1c380851e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f080       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0629c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8064b40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc14e8a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813ab20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026060       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00078a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805dc50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b45f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867eeb0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0008590       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98050230       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbfa0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135800       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38060c30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0e20       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b0800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c640399e0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cd350       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3dc0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38089ee0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c301da250       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80f1050       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc014bf0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a1a60       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100bc340       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304d0370       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812bf60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64051870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38094a60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38068320       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b84d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4037920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046890       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30000dc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbdd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c0e40       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00dfae0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90258ee0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804f6f0       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90045b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808b700       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8065550       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8060390       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5e40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e5cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc807f2c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805d8b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800adb40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400d760       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30326b80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a0c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029f4a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259b80       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00048c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00af830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8550       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811e7a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808e490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8129480       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1d14005160    2547  15675392  15603040  org.apache.flink.util.ChildFirstClassLoader
                                                                61    172032    106152   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1c380ea520       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857a720       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac004710       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00df0f0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3861a300       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c6120       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d50c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00020d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026190       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc009490       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d5510       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x0000559cae5496b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38081400       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7110       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8052210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64052a10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc008aa0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c1930       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134380       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d770       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc337240       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c58016fd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140140       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017460       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98059040       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c38674d70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38013c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30072cb0       1      6144      4032  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0005930       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b4ae0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812fb30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38032ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0166a0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000b9f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b8ca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b48d0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf00647f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053f00       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d58d73980       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080036f648  0x000000080036ef80  0x00007f1d5822a1b0    4979  34353152  34252632  jdk.internal.loader.ClassLoaders$AppClassLoader
                                                               564   2703360   1908736   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804fc70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bf800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402f6c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808efa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cc960       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380b0820       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6003d940       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3863da40       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38078d80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0060       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1ccc059790    2648  16330752  16287960  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4013060       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cb480       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7b60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3054d770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a8e68  0x000000080036f648  0x00007f1c60007a50       1      6144      3024  sun.reflect.misc.MethodUtil
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00de270       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca0028160       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d6b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d4c80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38077ab0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5660       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b8010       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f59c10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac010bb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bd6c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b53e0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400db60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1404e940       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc0147d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3430       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98093270       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8116dc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8aa50       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e49d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000f340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8113720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d21d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88072820       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808f610       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d67e0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80883a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0134f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3868db90       1      6144      4520  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303249f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807aff0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e08d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c045720       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c902475f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000fbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2cd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81297a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00abd60       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134860       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000a6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303246d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38099a70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812f150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000c610       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900072d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cd4052fc0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380140b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257e30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bd50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38015370       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f7a0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813b5d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8136870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811a8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf402a5a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804ebf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980aecb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c4330       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80b7850       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084d80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017d20       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc804c7a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c68003140       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4033900       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d14042ba0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90006f20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f230       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7750       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac830       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38088e70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0350       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81378e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30325540       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5a281630       1      6144      4768  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813cbd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8860       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808a920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380899f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc803f350       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c6d90       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380db5b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d2130       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00adbc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81312a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000ad00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ca2c0       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00be210       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bb5b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c386785e0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cf0064e50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38035d90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2e60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d36c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc0002210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c385c6f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e53c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c3a80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x0000000000000000  0x0000000000000000  0x00007f1d58223a40    3735  16637952  16601232  <boot class loader>
                                                               364   1014784    658032   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90033cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b6a40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80a1520       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f38e10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0a00       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8120170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c04fb30       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100c9b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81205e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122f30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60045bc0       1      6144      3992  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc04a8f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8ce60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81346e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38079f60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad8c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc900       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011de0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804e7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7580       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3009d730       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a8e68  0x00007f1cc0011e00       1      3072      2176  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c90004d90    2626  16461824  16391336  org.apache.flink.util.ChildFirstClassLoader
                                                                61    177152    111624   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc05d1f0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380e8400       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8130f80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c9820e410       1      6144      3848  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c38685650    2648  16330752  16286608  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8042c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81406e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90034980       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000bd60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80839f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38083d50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257460       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3803d6f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98074170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bb950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38096730       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9bd0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c780016c0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e27c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38087f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3fe0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c047930       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303d77a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bf60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38063580       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bdf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004e8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5980       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c4570       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812b740       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00074b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1560       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134b70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad5b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bcbd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4016dc0       1      3072      2192  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d99c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b7500       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ad230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38051d00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137a20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba3d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980700f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c386121b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029fe30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c3801e590    2651  16408576  16320192  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac470       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0009c00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88061f30       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c580154c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807be20       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139df0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5bd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8142d80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137710       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60056030       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1990       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74a40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38061f80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e05a0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bc0c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c981c06d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc046f80       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30328250       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60057560       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304fdc50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d450       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900222f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc001e850       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b6b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc335880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38138510       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857d030       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086290       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e04d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380870c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e0830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d5490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d75010       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc33bb70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c900333a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807cbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc019db0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c510       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000ba60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813a170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122d40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c600092b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d7610       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa1c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00062f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8131f90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c880739f0       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b1310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c20f0b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01b6b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc02fd70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811ac10       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc062900       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa4c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80bb720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9610       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8121e80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30327500       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e1ad0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080036ef80  0x0000000000000000  0x00007f1d582879b0     112    843776    783312  jdk.internal.loader.ClassLoaders$PlatformClassLoader
                                                                 3      9216      5432   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8080100       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80873d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b300       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc2e7b90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c350       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021200       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c210130       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8044950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4031cc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74ec0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba9e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011840       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00da8d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d5a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e57b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812e180       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8047410       1      6144      4008  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807a130       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8146e70       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
Total = 431                                                  23605  140124160  137261400  
ChunkSz: Total size of all allocated metaspace chunks
BlockSz: Total size of all allocated metaspace blocks (each chunk has several blocks)

 

There is a gentleman who posts blogs on Flink, and he goes into a bit more detail there. This was against Flink 1.9, but the foundational concepts still apply to 1.10. http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/

-K

On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <[hidden email]> wrote:

Another big potential candidate is the fact that JDBC libs I use in my job are put into the Flink lib folder instead of putting them into the fat jar..tomorrow I'll try to see if the metaspace is getting cleared correctly after that change.

Unfortunately our jobs were written before the child-first / parent-first classloading refactoring and at that time that was the way to go..but now it can cause this kind of problems if using child-first policy.

On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <[hidden email]> wrote:

Thank you Kye for your insights...in my mind, if the job runs without problems one or more times the heap size, and thus the medatadata-size, is big enough and I should not increase it (on the same data of course).

So I'll try to understand who is leaking what..the advice to avoid the dynamic class loading is just a workaround to me..there's something wrong going on and tomorrow I'll try to understand the root cause of the problem using -XX:NativeMemoryTracking=summary as you suggested.

I'll keep you up to date with my findings..

Best,

Flavio

On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <[hidden email]> wrote:

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

[Kye Bae]

I see:

                            (  Metadata:   )
                            (    reserved=122880KB, committed=121168KB)
                            (    used=117529KB)
                            (    free=3639KB)
                            (    waste=0KB =0.00%)

                          Internal (reserved=855KB, committed=855KB)
                            (malloc=823KB #2442)
                            (mmap: reserved=32KB, committed=32KB)

and

0x00000008405a5288  0x000000080036f648  0x00007f1d14005160    2547  15675392  15603040  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1ccc059790    2648  16330752  16287960  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c90004d90    2626  16461824  16391336  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c38685650    2648  16330752  16286608  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c3801e590    2651  16408576  16320192  org.apache.flink.util.ChildFirstClassLoader

The "Metadata" looks like there is a little over 3.5MB left. Not sure if this is enough. Also, "Internal" memory reserved is less than 1 MB, but this is where the Java direct memory buffers get allocated (Java NIO). Don't know the details of your Flink job, but is this enough?

Flink 1.10 memory architecture requires care, and there should be enough for both Flink and JVM internal operations. It shouldn't hurt to review your memory requirements and allocate extra to account for unexpected spikes. 

Please keep in mind Yarn will kill a container immediately if the physical memory limit is breached. 

Also, it looks like ChildFirstClassLoader (user-code loader - dynamic) was used 5 times to load classes onto this taskmanager. Were there 5 job submissions to your standalone cluster or 4 job restarts for the same job? Additional (dynamic) loadings will likely result in exceeding the max metaspace limit.

If your Flink cluster is meant to host a single job, I would avoid using the dynamic classloading altogether and then recapture the stats to see if there are still multiple loading of the user (your application) classes. If it doesn't, you might want to make that change to not use the dynamic classloading.

-K

On Tue, Nov 17, 2020 at 2:00 PM Flavio Pompermaier <[hidden email]> wrote:

Hi Kye, thanks a lot for the support! See my answers inline below

On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <[hidden email]> wrote:

It is possible, but I am not entirely sure about the load order affecting the metaspace usage.

To find out why your taskmanager container is exceeding the metaspace, we would need to know what value the max metaspace size is set to and then find out how much of the metaspace is actually being used and what the classloading stats are (before Yarn kills the container). 

If possible, please share the current metaspace setting and the native memory stats from one of your taskmanager instances (Java process as a Yarn container). I think those run as the yarn user as opposed to the hadoop user (it depends on your environment).

 

Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize is automatically computed and it is set to 256 MB (-XX:MaxMetaspaceSize=268435456)

# enable the native memory tracking referenced earlier - this may need to be passed in as the taskmanager JVM option parameter not the jobmanager.

# log onto one of the taskmanager HW instances

sudo su - yarn # or log on as the user that runs Yarn

jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner - this would be a taskmanager

jcmd TM_PID VM.native_memory summary # size information for "Class" - metaspace

 jcmd 6976 VM.native_memory summary
6976:

Native Memory Tracking:

Total: reserved=1473311KB, committed=954707KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB)
 
-                     Class (reserved=379640KB, committed=142024KB)
                            (classes #23605)
                            (  instance classes #22058, array classes #1547)
                            (malloc=2808KB #48655)
                            (mmap: reserved=376832KB, committed=139216KB)
                            (  Metadata:   )
                            (    reserved=122880KB, committed=121168KB)
                            (    used=117529KB)
                            (    free=3639KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=253952KB, committed=18048KB)
                            (    used=16516KB)
                            (    free=1532KB)
                            (    waste=0KB =0.00%)
 
-                    Thread (reserved=76359KB, committed=6867KB)
                            (thread #74)
                            (stack: reserved=76012KB, committed=6520KB)
                            (malloc=262KB #446)
                            (arena=85KB #146)
 
-                      Code (reserved=250407KB, committed=38911KB)
                            (malloc=2719KB #11989)
                            (mmap: reserved=247688KB, committed=36192KB)
 
-                        GC (reserved=71481KB, committed=71481KB)
                            (malloc=19121KB #17323)
                            (mmap: reserved=52360KB, committed=52360KB)
 
-                  Compiler (reserved=561KB, committed=561KB)
                            (malloc=429KB #846)
                            (arena=133KB #5)
 
-                  Internal (reserved=855KB, committed=855KB)
                            (malloc=823KB #2442)
                            (mmap: reserved=32KB, committed=32KB)
 
-                     Other (reserved=131857KB, committed=131857KB)
                            (malloc=131857KB #4128)
 
-                    Symbol (reserved=15702KB, committed=15702KB)
                            (malloc=14288KB #165850)
                            (arena=1415KB #1)
 
-    Native Memory Tracking (reserved=4072KB, committed=4072KB)
                            (malloc=27KB #353)
                            (tracking overhead=4045KB)
 
-        Shared class space (reserved=17044KB, committed=17044KB)
                            (mmap: reserved=17044KB, committed=17044KB)
 
-               Arena Chunk (reserved=199KB, committed=199KB)
                            (malloc=199KB)
 
-                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #191)
 
-                 Arguments (reserved=20KB, committed=20KB)
                            (malloc=20KB #501)
 
-                    Module (reserved=568KB, committed=568KB)
                            (malloc=568KB #3919)
 
-              Synchronizer (reserved=244KB, committed=244KB)
                            (malloc=244KB #2054)
 
-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

 

jcmd TM_PID VM.classloader_stats # how many classes were loaded by which classloader, sizes, etc.

 jcmd 6976 VM.classloader_stats
6976:
ClassLoader         Parent              CLD*               Classes   ChunkSz   BlockSz  Type
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d8140       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38193d50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00a9f10       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ab290       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38011150       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5d90       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004dec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ae230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402a2f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c5803b430       1      4096      3232  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c072ce0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800b7ee0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc336610       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084aa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc0011ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2f70       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021090       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8085cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d3830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00becf0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812a7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c880       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc018030       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139a90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90020cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9806f6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804c2d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80bcf80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc347ec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01aca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38576590       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380f71f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc40110d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c030       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2440       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c8003c3a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bb00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900cee70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053d30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b5a00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e29b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053250       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60057820       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867f350       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d765c0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc4014c00       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2f90       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae450       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc014ae0       1     10240      7392  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813e880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036ef80  0x00007f1c380851e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f080       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0629c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8064b40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc14e8a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813ab20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026060       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00078a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805dc50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b45f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867eeb0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0008590       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98050230       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbfa0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135800       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38060c30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0e20       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b0800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c640399e0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cd350       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3dc0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38089ee0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c301da250       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80f1050       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc014bf0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a1a60       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100bc340       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304d0370       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812bf60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64051870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38094a60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38068320       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b84d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4037920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046890       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30000dc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbdd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c0e40       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00dfae0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90258ee0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804f6f0       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90045b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808b700       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8065550       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8060390       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5e40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e5cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc807f2c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805d8b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800adb40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400d760       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30326b80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a0c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029f4a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259b80       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00048c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00af830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8550       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811e7a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808e490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8129480       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1d14005160    2547  15675392  15603040  org.apache.flink.util.ChildFirstClassLoader
                                                                61    172032    106152   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1c380ea520       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857a720       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac004710       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00df0f0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3861a300       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c6120       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d50c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00020d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026190       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc009490       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d5510       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x0000559cae5496b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38081400       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7110       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8052210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64052a10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc008aa0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c1930       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134380       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d770       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc337240       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c58016fd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140140       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017460       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98059040       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c38674d70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38013c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30072cb0       1      6144      4032  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0005930       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b4ae0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812fb30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38032ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0166a0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000b9f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b8ca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b48d0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf00647f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053f00       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d58d73980       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080036f648  0x000000080036ef80  0x00007f1d5822a1b0    4979  34353152  34252632  jdk.internal.loader.ClassLoaders$AppClassLoader
                                                               564   2703360   1908736   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804fc70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bf800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402f6c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808efa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cc960       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380b0820       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6003d940       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3863da40       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38078d80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0060       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1ccc059790    2648  16330752  16287960  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4013060       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cb480       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7b60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3054d770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a8e68  0x000000080036f648  0x00007f1c60007a50       1      6144      3024  sun.reflect.misc.MethodUtil
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00de270       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca0028160       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d6b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d4c80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38077ab0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5660       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b8010       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f59c10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac010bb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bd6c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b53e0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400db60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1404e940       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc0147d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3430       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98093270       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8116dc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8aa50       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e49d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000f340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8113720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d21d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88072820       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808f610       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d67e0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80883a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0134f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3868db90       1      6144      4520  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303249f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807aff0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e08d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c045720       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c902475f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000fbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2cd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81297a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00abd60       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134860       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000a6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303246d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38099a70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812f150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000c610       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900072d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cd4052fc0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380140b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257e30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bd50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38015370       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f7a0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813b5d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8136870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811a8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf402a5a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804ebf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980aecb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c4330       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80b7850       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084d80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017d20       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc804c7a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c68003140       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4033900       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d14042ba0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90006f20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f230       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7750       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac830       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38088e70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0350       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81378e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30325540       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5a281630       1      6144      4768  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813cbd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8860       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808a920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380899f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc803f350       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c6d90       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380db5b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d2130       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00adbc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81312a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000ad00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ca2c0       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00be210       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bb5b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c386785e0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cf0064e50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38035d90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2e60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d36c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc0002210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c385c6f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e53c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c3a80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x0000000000000000  0x0000000000000000  0x00007f1d58223a40    3735  16637952  16601232  <boot class loader>
                                                               364   1014784    658032   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90033cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b6a40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80a1520       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f38e10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0a00       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8120170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c04fb30       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100c9b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81205e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122f30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60045bc0       1      6144      3992  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc04a8f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8ce60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81346e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38079f60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad8c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc900       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011de0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804e7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7580       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3009d730       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a8e68  0x00007f1cc0011e00       1      3072      2176  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c90004d90    2626  16461824  16391336  org.apache.flink.util.ChildFirstClassLoader
                                                                61    177152    111624   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc05d1f0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380e8400       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8130f80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c9820e410       1      6144      3848  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c38685650    2648  16330752  16286608  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8042c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81406e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90034980       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000bd60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80839f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38083d50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257460       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3803d6f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98074170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bb950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38096730       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9bd0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c780016c0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e27c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38087f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3fe0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c047930       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303d77a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bf60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38063580       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bdf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004e8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5980       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c4570       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812b740       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00074b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1560       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134b70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad5b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bcbd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4016dc0       1      3072      2192  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d99c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b7500       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ad230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38051d00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137a20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba3d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980700f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c386121b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029fe30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c3801e590    2651  16408576  16320192  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac470       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0009c00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88061f30       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c580154c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807be20       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139df0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5bd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8142d80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137710       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60056030       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1990       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74a40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38061f80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e05a0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bc0c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c981c06d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc046f80       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30328250       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60057560       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304fdc50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d450       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900222f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc001e850       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b6b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc335880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38138510       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857d030       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086290       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e04d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380870c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e0830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d5490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d75010       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc33bb70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c900333a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807cbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc019db0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c510       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000ba60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813a170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122d40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c600092b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d7610       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa1c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00062f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8131f90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c880739f0       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b1310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c20f0b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01b6b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc02fd70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811ac10       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc062900       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa4c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80bb720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9610       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8121e80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30327500       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e1ad0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080036ef80  0x0000000000000000  0x00007f1d582879b0     112    843776    783312  jdk.internal.loader.ClassLoaders$PlatformClassLoader
                                                                 3      9216      5432   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8080100       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80873d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b300       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc2e7b90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c350       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021200       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c210130       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8044950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4031cc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74ec0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba9e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011840       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00da8d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d5a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e57b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812e180       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8047410       1      6144      4008  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807a130       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8146e70       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
Total = 431                                                  23605  140124160  137261400  
ChunkSz: Total size of all allocated metaspace chunks
BlockSz: Total size of all allocated metaspace blocks (each chunk has several blocks)

 

There is a gentleman who posts blogs on Flink, and he goes into a bit more detail there. This was against Flink 1.9, but the foundational concepts still apply to 1.10. http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/

-K

On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <[hidden email]> wrote:

Another big potential candidate is the fact that JDBC libs I use in my job are put into the Flink lib folder instead of putting them into the fat jar..tomorrow I'll try to see if the metaspace is getting cleared correctly after that change.

Unfortunately our jobs were written before the child-first / parent-first classloading refactoring and at that time that was the way to go..but now it can cause this kind of problems if using child-first policy.

On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <[hidden email]> wrote:

Thank you Kye for your insights...in my mind, if the job runs without problems one or more times the heap size, and thus the medatadata-size, is big enough and I should not increase it (on the same data of course).

So I'll try to understand who is leaking what..the advice to avoid the dynamic class loading is just a workaround to me..there's something wrong going on and tomorrow I'll try to understand the root cause of the problem using -XX:NativeMemoryTracking=summary as you suggested.

I'll keep you up to date with my findings..

Best,

Flavio

On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <[hidden email]> wrote:

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

[Flavio Pompermaier]

Hi Kye,

thank you a lot for your help.

There are a couple of things that are not clear to me:  in an ideal world, ChildFirstClassLoader instances should all be garbage collected right?

In my case they are not because they have some pointers to some parent class loader for some class..am I right?

Debugging a little bit the internals of the ChildFirstClass.loadClassWithoutExceptionHandling I saw that there are a couple of classes that are demanded to the parent class loader (i.e. the super.loadClassWithoutExceptionHandling() is called because a class is not found in the user classloader in the snippet at [1]):

• javax.security.*
• sun.security.krb5.Config
• jdk.internal.reflect.*
• javax.net.*
• sun.misc.unsafe
• com.microsoft.sqlserver.jdbc.*

While the latter could be moved into the fat jar because it is the dependency of the JDBC connector, the other classes are correctly loaded from the system class loader..

Is this stopping the user classloader to be correctly garbage collected? 

And a final question about the possible workaround to put the user jar into the Flink lib: in this case I don't have a jar to pass as argument to the CLI java client..is this a problem of the CLI client or not..? 

Should the run action be callable without specifying a user JAR but just the main class (in the case that it is in the lib dir)?

[1]

try {
     // check the URLs

     c = findClass(name);
} catch (ClassNotFoundException e) {
    // let URLClassLoader do it, which will eventually call the parent
    c = super.loadClassWithoutExceptionHandling(name, resolve);
}

On Tue, Nov 17, 2020 at 8:51 PM Kye Bae <[hidden email]> wrote:

I see:

                            (  Metadata:   )
                            (    reserved=122880KB, committed=121168KB)
                            (    used=117529KB)
                            (    free=3639KB)
                            (    waste=0KB =0.00%)

                          Internal (reserved=855KB, committed=855KB)
                            (malloc=823KB #2442)
                            (mmap: reserved=32KB, committed=32KB)

and

0x00000008405a5288  0x000000080036f648  0x00007f1d14005160    2547  15675392  15603040  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1ccc059790    2648  16330752  16287960  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c90004d90    2626  16461824  16391336  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c38685650    2648  16330752  16286608  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c3801e590    2651  16408576  16320192  org.apache.flink.util.ChildFirstClassLoader

The "Metadata" looks like there is a little over 3.5MB left. Not sure if this is enough. Also, "Internal" memory reserved is less than 1 MB, but this is where the Java direct memory buffers get allocated (Java NIO). Don't know the details of your Flink job, but is this enough?

Flink 1.10 memory architecture requires care, and there should be enough for both Flink and JVM internal operations. It shouldn't hurt to review your memory requirements and allocate extra to account for unexpected spikes. 

Please keep in mind Yarn will kill a container immediately if the physical memory limit is breached. 

Also, it looks like ChildFirstClassLoader (user-code loader - dynamic) was used 5 times to load classes onto this taskmanager. Were there 5 job submissions to your standalone cluster or 4 job restarts for the same job? Additional (dynamic) loadings will likely result in exceeding the max metaspace limit.

If your Flink cluster is meant to host a single job, I would avoid using the dynamic classloading altogether and then recapture the stats to see if there are still multiple loading of the user (your application) classes. If it doesn't, you might want to make that change to not use the dynamic classloading.

-K

On Tue, Nov 17, 2020 at 2:00 PM Flavio Pompermaier <[hidden email]> wrote:

Hi Kye, thanks a lot for the support! See my answers inline below

On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <[hidden email]> wrote:

It is possible, but I am not entirely sure about the load order affecting the metaspace usage.

To find out why your taskmanager container is exceeding the metaspace, we would need to know what value the max metaspace size is set to and then find out how much of the metaspace is actually being used and what the classloading stats are (before Yarn kills the container). 

If possible, please share the current metaspace setting and the native memory stats from one of your taskmanager instances (Java process as a Yarn container). I think those run as the yarn user as opposed to the hadoop user (it depends on your environment).

 

Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize is automatically computed and it is set to 256 MB (-XX:MaxMetaspaceSize=268435456)

# enable the native memory tracking referenced earlier - this may need to be passed in as the taskmanager JVM option parameter not the jobmanager.

# log onto one of the taskmanager HW instances

sudo su - yarn # or log on as the user that runs Yarn

jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner - this would be a taskmanager

jcmd TM_PID VM.native_memory summary # size information for "Class" - metaspace

 jcmd 6976 VM.native_memory summary
6976:

Native Memory Tracking:

Total: reserved=1473311KB, committed=954707KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB)
 
-                     Class (reserved=379640KB, committed=142024KB)
                            (classes #23605)
                            (  instance classes #22058, array classes #1547)
                            (malloc=2808KB #48655)
                            (mmap: reserved=376832KB, committed=139216KB)
                            (  Metadata:   )
                            (    reserved=122880KB, committed=121168KB)
                            (    used=117529KB)
                            (    free=3639KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=253952KB, committed=18048KB)
                            (    used=16516KB)
                            (    free=1532KB)
                            (    waste=0KB =0.00%)
 
-                    Thread (reserved=76359KB, committed=6867KB)
                            (thread #74)
                            (stack: reserved=76012KB, committed=6520KB)
                            (malloc=262KB #446)
                            (arena=85KB #146)
 
-                      Code (reserved=250407KB, committed=38911KB)
                            (malloc=2719KB #11989)
                            (mmap: reserved=247688KB, committed=36192KB)
 
-                        GC (reserved=71481KB, committed=71481KB)
                            (malloc=19121KB #17323)
                            (mmap: reserved=52360KB, committed=52360KB)
 
-                  Compiler (reserved=561KB, committed=561KB)
                            (malloc=429KB #846)
                            (arena=133KB #5)
 
-                  Internal (reserved=855KB, committed=855KB)
                            (malloc=823KB #2442)
                            (mmap: reserved=32KB, committed=32KB)
 
-                     Other (reserved=131857KB, committed=131857KB)
                            (malloc=131857KB #4128)
 
-                    Symbol (reserved=15702KB, committed=15702KB)
                            (malloc=14288KB #165850)
                            (arena=1415KB #1)
 
-    Native Memory Tracking (reserved=4072KB, committed=4072KB)
                            (malloc=27KB #353)
                            (tracking overhead=4045KB)
 
-        Shared class space (reserved=17044KB, committed=17044KB)
                            (mmap: reserved=17044KB, committed=17044KB)
 
-               Arena Chunk (reserved=199KB, committed=199KB)
                            (malloc=199KB)
 
-                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #191)
 
-                 Arguments (reserved=20KB, committed=20KB)
                            (malloc=20KB #501)
 
-                    Module (reserved=568KB, committed=568KB)
                            (malloc=568KB #3919)
 
-              Synchronizer (reserved=244KB, committed=244KB)
                            (malloc=244KB #2054)
 
-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

 

jcmd TM_PID VM.classloader_stats # how many classes were loaded by which classloader, sizes, etc.

 jcmd 6976 VM.classloader_stats
6976:
ClassLoader         Parent              CLD*               Classes   ChunkSz   BlockSz  Type
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d8140       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38193d50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00a9f10       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ab290       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38011150       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5d90       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004dec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ae230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402a2f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c5803b430       1      4096      3232  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c072ce0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800b7ee0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc336610       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084aa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc0011ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2f70       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021090       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8085cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d3830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00becf0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812a7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c880       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc018030       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139a90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90020cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9806f6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804c2d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80bcf80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc347ec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01aca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38576590       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380f71f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc40110d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c030       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2440       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c8003c3a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bb00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900cee70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053d30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b5a00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e29b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053250       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60057820       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867f350       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d765c0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc4014c00       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2f90       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae450       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc014ae0       1     10240      7392  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813e880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036ef80  0x00007f1c380851e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f080       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0629c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8064b40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc14e8a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813ab20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026060       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00078a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805dc50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b45f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867eeb0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0008590       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98050230       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbfa0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135800       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38060c30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0e20       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b0800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c640399e0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cd350       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3dc0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38089ee0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c301da250       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80f1050       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc014bf0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a1a60       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100bc340       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304d0370       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812bf60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64051870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38094a60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38068320       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b84d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4037920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046890       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30000dc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbdd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c0e40       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00dfae0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90258ee0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804f6f0       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90045b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808b700       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8065550       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8060390       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5e40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e5cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc807f2c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805d8b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800adb40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400d760       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30326b80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a0c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029f4a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259b80       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00048c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00af830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8550       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811e7a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808e490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8129480       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1d14005160    2547  15675392  15603040  org.apache.flink.util.ChildFirstClassLoader
                                                                61    172032    106152   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1c380ea520       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857a720       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac004710       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00df0f0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3861a300       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c6120       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d50c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00020d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026190       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc009490       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d5510       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x0000559cae5496b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38081400       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7110       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8052210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64052a10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc008aa0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c1930       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134380       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d770       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc337240       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c58016fd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140140       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017460       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98059040       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c38674d70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38013c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30072cb0       1      6144      4032  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0005930       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b4ae0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812fb30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38032ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0166a0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000b9f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b8ca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b48d0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf00647f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053f00       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d58d73980       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080036f648  0x000000080036ef80  0x00007f1d5822a1b0    4979  34353152  34252632  jdk.internal.loader.ClassLoaders$AppClassLoader
                                                               564   2703360   1908736   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804fc70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bf800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402f6c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808efa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cc960       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380b0820       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6003d940       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3863da40       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38078d80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0060       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1ccc059790    2648  16330752  16287960  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4013060       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cb480       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7b60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3054d770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a8e68  0x000000080036f648  0x00007f1c60007a50       1      6144      3024  sun.reflect.misc.MethodUtil
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00de270       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca0028160       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d6b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d4c80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38077ab0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5660       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b8010       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f59c10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac010bb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bd6c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b53e0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400db60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1404e940       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc0147d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3430       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98093270       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8116dc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8aa50       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e49d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000f340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8113720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d21d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88072820       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808f610       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d67e0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80883a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0134f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3868db90       1      6144      4520  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303249f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807aff0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e08d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c045720       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c902475f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000fbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2cd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81297a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00abd60       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134860       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000a6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303246d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38099a70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812f150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000c610       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900072d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cd4052fc0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380140b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257e30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bd50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38015370       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f7a0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813b5d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8136870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811a8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf402a5a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804ebf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980aecb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c4330       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80b7850       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084d80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017d20       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc804c7a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c68003140       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4033900       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d14042ba0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90006f20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f230       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7750       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac830       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38088e70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0350       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81378e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30325540       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5a281630       1      6144      4768  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813cbd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8860       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808a920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380899f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc803f350       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c6d90       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380db5b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d2130       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00adbc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81312a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000ad00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ca2c0       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00be210       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bb5b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c386785e0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cf0064e50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38035d90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2e60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d36c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc0002210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c385c6f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e53c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c3a80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x0000000000000000  0x0000000000000000  0x00007f1d58223a40    3735  16637952  16601232  <boot class loader>
                                                               364   1014784    658032   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90033cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b6a40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80a1520       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f38e10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0a00       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8120170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c04fb30       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100c9b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81205e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122f30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60045bc0       1      6144      3992  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc04a8f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8ce60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81346e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38079f60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad8c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc900       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011de0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804e7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7580       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3009d730       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a8e68  0x00007f1cc0011e00       1      3072      2176  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c90004d90    2626  16461824  16391336  org.apache.flink.util.ChildFirstClassLoader
                                                                61    177152    111624   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc05d1f0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380e8400       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8130f80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c9820e410       1      6144      3848  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c38685650    2648  16330752  16286608  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8042c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81406e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90034980       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000bd60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80839f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38083d50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257460       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3803d6f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98074170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bb950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38096730       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9bd0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c780016c0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e27c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38087f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3fe0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c047930       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303d77a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bf60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38063580       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bdf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004e8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5980       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c4570       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812b740       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00074b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1560       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134b70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad5b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bcbd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4016dc0       1      3072      2192  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d99c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b7500       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ad230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38051d00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137a20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba3d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980700f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c386121b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029fe30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c3801e590    2651  16408576  16320192  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac470       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0009c00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88061f30       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c580154c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807be20       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139df0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5bd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8142d80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137710       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60056030       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1990       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74a40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38061f80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e05a0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bc0c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c981c06d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc046f80       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30328250       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60057560       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304fdc50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d450       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900222f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc001e850       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b6b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc335880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38138510       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857d030       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086290       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e04d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380870c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e0830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d5490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d75010       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc33bb70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c900333a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807cbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc019db0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c510       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000ba60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813a170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122d40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c600092b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d7610       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa1c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00062f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8131f90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c880739f0       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b1310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c20f0b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01b6b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc02fd70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811ac10       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc062900       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa4c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80bb720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9610       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8121e80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30327500       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e1ad0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080036ef80  0x0000000000000000  0x00007f1d582879b0     112    843776    783312  jdk.internal.loader.ClassLoaders$PlatformClassLoader
                                                                 3      9216      5432   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8080100       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80873d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b300       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc2e7b90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c350       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021200       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c210130       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8044950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4031cc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74ec0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba9e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011840       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00da8d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d5a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e57b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812e180       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8047410       1      6144      4008  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807a130       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8146e70       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
Total = 431                                                  23605  140124160  137261400  
ChunkSz: Total size of all allocated metaspace chunks
BlockSz: Total size of all allocated metaspace blocks (each chunk has several blocks)

 

There is a gentleman who posts blogs on Flink, and he goes into a bit more detail there. This was against Flink 1.9, but the foundational concepts still apply to 1.10. http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/

-K

On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <[hidden email]> wrote:

Another big potential candidate is the fact that JDBC libs I use in my job are put into the Flink lib folder instead of putting them into the fat jar..tomorrow I'll try to see if the metaspace is getting cleared correctly after that change.

Unfortunately our jobs were written before the child-first / parent-first classloading refactoring and at that time that was the way to go..but now it can cause this kind of problems if using child-first policy.

On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <[hidden email]> wrote:

Thank you Kye for your insights...in my mind, if the job runs without problems one or more times the heap size, and thus the medatadata-size, is big enough and I should not increase it (on the same data of course).

So I'll try to understand who is leaking what..the advice to avoid the dynamic class loading is just a workaround to me..there's something wrong going on and tomorrow I'll try to understand the root cause of the problem using -XX:NativeMemoryTracking=summary as you suggested.

I'll keep you up to date with my findings..

Best,

Flavio

On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <[hidden email]> wrote:

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

[Flavio Pompermaier]

Could this problem be related to https://issues.apache.org/jira/browse/FLINK-20333, where the instances of java.io.ObjectStreamClass$Caches need to be cleaned up as a fix of https://bugs.openjdk.java.net/browse/JDK-8199589?

On Wed, Nov 18, 2020 at 3:04 PM Flavio Pompermaier <[hidden email]> wrote:

Hi Kye,

thank you a lot for your help.

There are a couple of things that are not clear to me:  in an ideal world, ChildFirstClassLoader instances should all be garbage collected right?

In my case they are not because they have some pointers to some parent class loader for some class..am I right?

Debugging a little bit the internals of the ChildFirstClass.loadClassWithoutExceptionHandling I saw that there are a couple of classes that are demanded to the parent class loader (i.e. the super.loadClassWithoutExceptionHandling() is called because a class is not found in the user classloader in the snippet at [1]):

• javax.security.*
• sun.security.krb5.Config
• jdk.internal.reflect.*
• javax.net.*
• sun.misc.unsafe
• com.microsoft.sqlserver.jdbc.*

While the latter could be moved into the fat jar because it is the dependency of the JDBC connector, the other classes are correctly loaded from the system class loader..

Is this stopping the user classloader to be correctly garbage collected? 

And a final question about the possible workaround to put the user jar into the Flink lib: in this case I don't have a jar to pass as argument to the CLI java client..is this a problem of the CLI client or not..? 

Should the run action be callable without specifying a user JAR but just the main class (in the case that it is in the lib dir)?

[1]

try {
     // check the URLs

     c = findClass(name);
} catch (ClassNotFoundException e) {
    // let URLClassLoader do it, which will eventually call the parent
    c = super.loadClassWithoutExceptionHandling(name, resolve);
}

On Tue, Nov 17, 2020 at 8:51 PM Kye Bae <[hidden email]> wrote:

I see:

                            (  Metadata:   )
                            (    reserved=122880KB, committed=121168KB)
                            (    used=117529KB)
                            (    free=3639KB)
                            (    waste=0KB =0.00%)

                          Internal (reserved=855KB, committed=855KB)
                            (malloc=823KB #2442)
                            (mmap: reserved=32KB, committed=32KB)

and

0x00000008405a5288  0x000000080036f648  0x00007f1d14005160    2547  15675392  15603040  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1ccc059790    2648  16330752  16287960  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c90004d90    2626  16461824  16391336  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c38685650    2648  16330752  16286608  org.apache.flink.util.ChildFirstClassLoader

0x00000008405a5288  0x000000080036f648  0x00007f1c3801e590    2651  16408576  16320192  org.apache.flink.util.ChildFirstClassLoader

The "Metadata" looks like there is a little over 3.5MB left. Not sure if this is enough. Also, "Internal" memory reserved is less than 1 MB, but this is where the Java direct memory buffers get allocated (Java NIO). Don't know the details of your Flink job, but is this enough?

Flink 1.10 memory architecture requires care, and there should be enough for both Flink and JVM internal operations. It shouldn't hurt to review your memory requirements and allocate extra to account for unexpected spikes. 

Please keep in mind Yarn will kill a container immediately if the physical memory limit is breached. 

Also, it looks like ChildFirstClassLoader (user-code loader - dynamic) was used 5 times to load classes onto this taskmanager. Were there 5 job submissions to your standalone cluster or 4 job restarts for the same job? Additional (dynamic) loadings will likely result in exceeding the max metaspace limit.

If your Flink cluster is meant to host a single job, I would avoid using the dynamic classloading altogether and then recapture the stats to see if there are still multiple loading of the user (your application) classes. If it doesn't, you might want to make that change to not use the dynamic classloading.

-K

On Tue, Nov 17, 2020 at 2:00 PM Flavio Pompermaier <[hidden email]> wrote:

Hi Kye, thanks a lot for the support! See my answers inline below

On Tue, Nov 17, 2020 at 7:22 PM Kye Bae <[hidden email]> wrote:

It is possible, but I am not entirely sure about the load order affecting the metaspace usage.

To find out why your taskmanager container is exceeding the metaspace, we would need to know what value the max metaspace size is set to and then find out how much of the metaspace is actually being used and what the classloading stats are (before Yarn kills the container). 

If possible, please share the current metaspace setting and the native memory stats from one of your taskmanager instances (Java process as a Yarn container). I think those run as the yarn user as opposed to the hadoop user (it depends on your environment).

 

Actually I don't use YARN, I use standalone cluster and MaxMetspaceSize is automatically computed and it is set to 256 MB (-XX:MaxMetaspaceSize=268435456)

# enable the native memory tracking referenced earlier - this may need to be passed in as the taskmanager JVM option parameter not the jobmanager.

# log onto one of the taskmanager HW instances

sudo su - yarn # or log on as the user that runs Yarn

jps -v # grab the PID for one of the processes named YarnTaskExecutorRunner - this would be a taskmanager

jcmd TM_PID VM.native_memory summary # size information for "Class" - metaspace

 jcmd 6976 VM.native_memory summary
6976:

Native Memory Tracking:

Total: reserved=1473311KB, committed=954707KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB)
 
-                     Class (reserved=379640KB, committed=142024KB)
                            (classes #23605)
                            (  instance classes #22058, array classes #1547)
                            (malloc=2808KB #48655)
                            (mmap: reserved=376832KB, committed=139216KB)
                            (  Metadata:   )
                            (    reserved=122880KB, committed=121168KB)
                            (    used=117529KB)
                            (    free=3639KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=253952KB, committed=18048KB)
                            (    used=16516KB)
                            (    free=1532KB)
                            (    waste=0KB =0.00%)
 
-                    Thread (reserved=76359KB, committed=6867KB)
                            (thread #74)
                            (stack: reserved=76012KB, committed=6520KB)
                            (malloc=262KB #446)
                            (arena=85KB #146)
 
-                      Code (reserved=250407KB, committed=38911KB)
                            (malloc=2719KB #11989)
                            (mmap: reserved=247688KB, committed=36192KB)
 
-                        GC (reserved=71481KB, committed=71481KB)
                            (malloc=19121KB #17323)
                            (mmap: reserved=52360KB, committed=52360KB)
 
-                  Compiler (reserved=561KB, committed=561KB)
                            (malloc=429KB #846)
                            (arena=133KB #5)
 
-                  Internal (reserved=855KB, committed=855KB)
                            (malloc=823KB #2442)
                            (mmap: reserved=32KB, committed=32KB)
 
-                     Other (reserved=131857KB, committed=131857KB)
                            (malloc=131857KB #4128)
 
-                    Symbol (reserved=15702KB, committed=15702KB)
                            (malloc=14288KB #165850)
                            (arena=1415KB #1)
 
-    Native Memory Tracking (reserved=4072KB, committed=4072KB)
                            (malloc=27KB #353)
                            (tracking overhead=4045KB)
 
-        Shared class space (reserved=17044KB, committed=17044KB)
                            (mmap: reserved=17044KB, committed=17044KB)
 
-               Arena Chunk (reserved=199KB, committed=199KB)
                            (malloc=199KB)
 
-                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB #191)
 
-                 Arguments (reserved=20KB, committed=20KB)
                            (malloc=20KB #501)
 
-                    Module (reserved=568KB, committed=568KB)
                            (malloc=568KB #3919)
 
-              Synchronizer (reserved=244KB, committed=244KB)
                            (malloc=244KB #2054)
 
-                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)

 

jcmd TM_PID VM.classloader_stats # how many classes were loaded by which classloader, sizes, etc.

 jcmd 6976 VM.classloader_stats
6976:
ClassLoader         Parent              CLD*               Classes   ChunkSz   BlockSz  Type
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d8140       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38193d50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00a9f10       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ab290       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38011150       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5d90       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004dec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ae230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402a2f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c5803b430       1      4096      3232  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c072ce0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800b7ee0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc336610       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084aa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc0011ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2f70       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021090       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8085cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d3830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00becf0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812a7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c880       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc018030       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139a90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90020cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9806f6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804c2d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80bcf80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc347ec0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01aca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38576590       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380f71f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc40110d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c030       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c2440       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c8003c3a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bb00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900cee70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053d30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b5a00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e29b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053250       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60057820       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867f350       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d765c0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc4014c00       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2f90       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae450       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc014ae0       1     10240      7392  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813e880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036ef80  0x00007f1c380851e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811f080       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0629c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8064b40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc14e8a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813ab20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026060       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00078a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805dc50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b45f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3867eeb0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0008590       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98050230       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbfa0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135800       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38060c30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0e20       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b0800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c640399e0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cd350       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3dc0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38089ee0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c98053b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c301da250       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80f1050       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc014bf0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a1a60       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100bc340       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304d0370       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812bf60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64051870       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38094a60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38068320       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b84d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4037920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046890       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30000dc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cbdd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c0e40       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00dfae0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90258ee0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804f6f0       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90045b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808b700       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8065550       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8060390       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b5e40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e5cf0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc807f2c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc805d8b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c800adb40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400d760       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30326b80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a0c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029f4a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90259b80       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00048c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00af830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8550       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811e7a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808e490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8129480       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1d14005160    2547  15675392  15603040  org.apache.flink.util.ChildFirstClassLoader
                                                                61    172032    106152   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1c380ea520       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857a720       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac004710       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00df0f0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3861a300       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c6120       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d50c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00020d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c64026190       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc009490       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d5510       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x0000559cae5496b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38081400       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7110       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8052210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c64052a10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc008aa0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c1930       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134380       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d770       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc337240       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c58016fd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140140       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017460       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98059040       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c38674d70       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38013c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30072cb0       1      6144      4032  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0005930       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b4ae0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812fb30       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38032ba0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0166a0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000b9f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80b8ca0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b48d0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf00647f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8053f00       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d58d73980       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080036f648  0x000000080036ef80  0x00007f1d5822a1b0    4979  34353152  34252632  jdk.internal.loader.ClassLoaders$AppClassLoader
                                                               564   2703360   1908736   + unsafe anonymous classes
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804fc70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bf800       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6402f6c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808efa0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cc960       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380b0820       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c6003d940       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3863da40       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8135b60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38078d80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0060       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1ccc059790    2648  16330752  16287960  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4013060       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00cb480       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7b60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3054d770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a8e68  0x000000080036f648  0x00007f1c60007a50       1      6144      3024  sun.reflect.misc.MethodUtil
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00de270       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca0028160       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d6b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d4c80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38077ab0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5660       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b8010       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f59c10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cac010bb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bd6c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b53e0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc400db60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1404e940       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc0147d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b3430       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98093270       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc7c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8116dc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8aa50       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e49d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000f340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cb8113720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d21d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88072820       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3808f610       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d67e0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80883a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc0134f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3868db90       1      6144      4520  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303249f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807aff0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e08d0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c045720       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c902475f0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000fbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2cd0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81297a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00abd60       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134860       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000a6c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303246d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38099a70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812f150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000c610       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900072d0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cd4052fc0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380140b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257e30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bd50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38015370       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f7a0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813b5d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8136870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811a8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cf402a5a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804ebf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980aecb0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80c4330       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80b7850       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8084d80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc017d20       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc804c7a0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c68003140       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4033900       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d14042ba0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8046210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90006f20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5857f230       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7750       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac830       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38088e70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0350       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81378e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c30325540       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d5a281630       1      6144      4768  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813cbd0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c8860       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c3808a920       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380899f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc803f350       1      6144      4488  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c6d90       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380db5b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d2130       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00adbc0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81312a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000ad00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ca2c0       1      6144      4464  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00be210       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bb5b0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c386785e0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cf0064e50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38035d90       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e2e60       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d36c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc0002210       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c385c6f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e53c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c3a80       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x0000000000000000  0x0000000000000000  0x00007f1d58223a40    3735  16637952  16601232  <boot class loader>
                                                               364   1014784    658032   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90033cd0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b6a40       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80a1520       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58f38e10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d0a00       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8120170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1d1c04fb30       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c100c9b10       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81205e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122f30       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c60045bc0       1      6144      3992  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc04a8f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d8ce60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3930       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81346e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38079f60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad8c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bc900       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011de0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c9804e7f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c7580       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3009d730       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a8e68  0x00007f1cc0011e00       1      3072      2176  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c90004d90    2626  16461824  16391336  org.apache.flink.util.ChildFirstClassLoader
                                                                61    177152    111624   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c0310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc05d1f0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c380e8400       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8130f80       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c9820e410       1      6144      3848  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c38685650    2648  16330752  16286608  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8042c50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb81406e0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90034980       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000bd60       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80839f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38083d50       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90257460       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3803d6f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c98074170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380bb950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38096730       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9bd0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c780016c0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e27c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38087f00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086150       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e3fe0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1d1c047930       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c303d77a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cd402bf60       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38063580       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9816bdf0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9004e8c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5980       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c4570       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812b740       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00074b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1560       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8134b70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ad5b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bcbd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc4016dc0       1      3072      2192  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d99c0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b7500       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380ad230       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38051d00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137a20       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba3d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980700f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c386121b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ae770       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9029fe30       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x00000008405a5288  0x000000080036f648  0x00007f1c3801e590    2651  16408576  16320192  org.apache.flink.util.ChildFirstClassLoader
                                                                61    175104    109552   + unsafe anonymous classes
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00ac470       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc0009c00       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c88061f30       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c580154c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807be20       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8139df0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00c5bd0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8142d80       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8137710       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60056030       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380c1990       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c902a0420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74a40       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38061f80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80e05a0       1      4096      3160  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00bc0c0       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c981c06d0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc046f80       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30328250       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc804a340       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c60057560       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c304fdc50       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812d450       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c900222f0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cc001e850       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b6b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc335880       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c38138510       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c3857d030       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c38086290       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e04d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c380870c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e0830       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c980d5490       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d75010       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc33bb70       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c900333a0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807cbb0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc019db0       1      6144      4760  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc000c510       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc000ba60       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8140870       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813a170       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8122d40       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c600092b0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c381d7610       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa1c0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc00062f0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8131f90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1c880739f0       1      6144      4024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d7420       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00b1310       1      3072      2024  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c20f0b0       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc01b6b0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc02fd70       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb811ac10       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1ccc062900       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80aa4c0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80bb720       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00d9610       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8121e80       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c30327500       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80e1ad0       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080036ef80  0x0000000000000000  0x00007f1d582879b0     112    843776    783312  jdk.internal.loader.ClassLoaders$PlatformClassLoader
                                                                 3      9216      5432   + unsafe anonymous classes
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8080100       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc80873d0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc00b300       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1ccc2e7b90       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb813c350       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c90021200       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3c210130       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8044950       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ca4031cc0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1d58d74ec0       1      6144      3960  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc8081b00       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1cc80ba9e0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1ccc011840       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00da8d0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1c9000d5a0       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1bb00e57b0       1      6144      4048  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb812e180       1      6144      3952  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x0000000000000000  0x00007f1cc8047410       1      6144      4008  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x000000080036f648  0x00007f1c3807a130       1      6144      3872  jdk.internal.reflect.DelegatingClassLoader
0x000000080039f8b0  0x00000008405a5288  0x00007f1cb8146e70       1      6144      3928  jdk.internal.reflect.DelegatingClassLoader
Total = 431                                                  23605  140124160  137261400  
ChunkSz: Total size of all allocated metaspace chunks
BlockSz: Total size of all allocated metaspace blocks (each chunk has several blocks)

 

There is a gentleman who posts blogs on Flink, and he goes into a bit more detail there. This was against Flink 1.9, but the foundational concepts still apply to 1.10. http://cloudsqale.com/2020/04/29/flink-1-9-off-heap-memory-on-yarn-troubleshooting-container-is-running-beyond-physical-memory-limits-errors/

-K

On Tue, Nov 17, 2020 at 12:47 PM Flavio Pompermaier <[hidden email]> wrote:

Another big potential candidate is the fact that JDBC libs I use in my job are put into the Flink lib folder instead of putting them into the fat jar..tomorrow I'll try to see if the metaspace is getting cleared correctly after that change.

Unfortunately our jobs were written before the child-first / parent-first classloading refactoring and at that time that was the way to go..but now it can cause this kind of problems if using child-first policy.

On Mon, Nov 16, 2020 at 8:44 PM Flavio Pompermaier <[hidden email]> wrote:

Thank you Kye for your insights...in my mind, if the job runs without problems one or more times the heap size, and thus the medatadata-size, is big enough and I should not increase it (on the same data of course).

So I'll try to understand who is leaking what..the advice to avoid the dynamic class loading is just a workaround to me..there's something wrong going on and tomorrow I'll try to understand the root cause of the problem using -XX:NativeMemoryTracking=summary as you suggested.

I'll keep you up to date with my findings..

Best,

Flavio

On Mon, Nov 16, 2020 at 8:22 PM Kye Bae <[hidden email]> wrote:

Hello!

The JVM metaspace is where all the classes (not class instances or objects) get loaded. jmap -histo is going to show you the heap space usage info not the metaspace.

You could inspect what is happening in the metaspace by using jcmd (e.g., jcmd JPID VM.native_memory summary) after restarting the cluster with "-XX:NativeMemoryTracking=summary"

As the error message suggests, you may need to increase taskmanager.memory.jvm-metaspace.size, but you need to be slightly careful when specifying the memory parameters in flink-conf.yaml in Flink 1.10 (they have an issue with a confusing error message).

Anothing thing to keep in mind is that you may want to avoid using dynamic classloading (https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/debugging_classloading.html#avoiding-dynamic-classloading-for-user-code): when the job continuously fails for some temporary issues, it will load the same class files into the metaspace multiple times and could exceed whatever the limit you set it.

-K

On Mon, Nov 16, 2020 at 12:39 PM Jan Lukavský <[hidden email]> wrote:

The exclusions should not have any impact on that, because what defines which classloader will load which class is not the presence or particular class in a specific jar, but the configuration of parent-first-patterns [1].

If you don't use any flink internal imports, than it still might be the case, that a class in any of the packages defined by the parent-first-pattern to hold reference to your user-code classes, which would cause the leak. I'd recommend to inspect the heap dump after several restarts of the application and look for reference to Class objects from the root set.

Jan

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#class-loading

On 11/16/20 5:34 PM, Flavio Pompermaier wrote:

I've tried to remove all possible imports of classes not contained in the fat jar but I still face the same problem.

I've also tried to reduce as much as possible the exclude in the shade section of the maven plugin (I took the one at [1]) so now I exclude only few dependencies..could it be that I should include org.slf4j:* if I use static import of it?

<artifactSet>
    <excludes>
      <exclude>com.google.code.findbugs:jsr305</exclude>
      <exclude>org.slf4j:*</exclude>
      <exclude>log4j:*</exclude>
    </excludes>
</artifactSet>

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/project-configuration.html#appendix-template-for-building-a-jar-with-dependencies

On Mon, Nov 16, 2020 at 3:29 PM Jan Lukavský <[hidden email]> wrote:

Yes, that could definitely cause this. You should probably avoid using these flink-internal shaded classes and ship your own versions (not shaded).

Best,

 Jan

On 11/16/20 3:22 PM, Flavio Pompermaier wrote:

Thank you Jan for your valuable feedback.

Could it be that I should not use import shaded-jackson classes in my user code?

For example import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper?

Bets,

Flavio

On Mon, Nov 16, 2020 at 3:15 PM Jan Lukavský <[hidden email]> wrote:

Hi Flavio,

when I encountered quite similar problem that you describe, it was related to a static storage located in class that was loaded "parent-first". In my case it was it was in java.lang.ClassValue, but it might (and probably will be) different in your case. The problem is that if user-code registers something in some (static) storage located in class loaded with parent (TaskTracker) classloader, then its associated classes will never be GC'd and Metaspace will grow. A good starting point would be not to focus on biggest consumers of heap (in general), but to look at where the 15k objects of type Class are referenced from. That might help you figure this out. I'm not sure if there is something that can be done in general to prevent this type of leaks. That would be probably question on dev@ mailing list.

Best,

 Jan

On 11/16/20 2:27 PM, Flavio Pompermaier wrote:

Hello everybody,

I was writing this email when a similar thread on this mailing list appeared..

The difference is that the other problem seems to be related with Flink 1.10 on YARN and does not output anything helpful in debugging the cause of the problem.

Indeed, in my use case I use Flink 1.11.0 and Flink on a standalone session cluster (the job is submitted to the cluster using the CLI client).

The problem arises when I submit the same job for about 20 times (this number unfortunately is not deterministic and can change a little bit). The error reported by the Task Executor is related to the ever growing Metaspace..the error seems to be pretty detailed [1].

I found the same issue in some previous threads on this mailing list and I've tried to figure it out the cause of the problem. The issue is that looking at the objects allocated I don't really get an idea of the source of the problem because the type of objects that are consuming the memory are of general purpose (i.e. Bytes, Integers and Strings)...these are my "top" memory consumers if looking at the output of  jmap -histo <PID>:

At run 0:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         46238       13224056  [B ([hidden email])
   2:          3736        6536672  [I ([hidden email])
   3:         38081         913944  java.lang.String ([hidden email])
   4:            26         852384  [Lakka.dispatch.forkjoin.ForkJoinTask;
   5:          7146         844984  java.lang.Class ([hidden email])

At run 1:

   1:         77.608       25.317.496  [B ([hidden email])
   2:          7.004        9.088.360  [I ([hidden email])
   3:         15.814        1.887.256  java.lang.Class ([hidden email])
   4:         67.381        1.617.144  java.lang.String ([hidden email])
   5:          3.906        1.422.960  [Ljava.util.HashMap$Node; ([hidden email])

At run 6:

   1:         81.408       25.375.400  [B ([hidden email])
   2:         12.479        7.249.392  [I ([hidden email])
   3:         29.090        3.496.168  java.lang.Class ([hidden email])
   4:          4.347        2.813.416  [Ljava.util.HashMap$Node; ([hidden email])
   5:         71.584        1.718.016  java.lang.String ([hidden email])

At run 8:

   1:        985.979      127.193.256  [B ([hidden email])
   2:         35.400       13.702.112  [I ([hidden email])
   3:        260.387        6.249.288  java.lang.String ([hidden email])
   4:        148.836        5.953.440  java.util.HashMap$KeyIterator ([hidden email])
   5:         17.641        5.222.344  [Ljava.util.HashMap$Node; ([hidden email])

Thanks in advance for any help,

Flavio

[1] --------------------------------------------------------------------------------------------------

java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:550) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:458) ~[?:?]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:452) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:451) ~[?:?]
        at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
        at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) [flink-dist_2.12-1.11.0.jar:1.11.0]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:522) [?:?]

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

---

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.