(DEPRECATED) Apache Flink User Mailing List archive.

[Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

Classic

List

Threaded

4 messages Options

Niels Basjes

[Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

Hi,

I'm running a lot of batch jobs on Kubernetes once in a while I get this exception.
What is causing this?
How can I fix this?

Niels Basjes

java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:45)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:53)
    at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:104)
    at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(ByteSizeValue.java:39)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:88)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:80)
    at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor.java:174)

Best regards / Met vriendelijke groeten,

Niels Basjes

Yang Wang

Re: [Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

From 1.10, Flink will enable the metaspace limit via "-XX:MaxMetaspaceSize"

by default. The default value is 96m, loading too many classes will cause

"OutOfMemoryError: Metaspace"[1]. So you need to increase the configured value.

[1]. https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_trouble.html#outofmemoryerror-metaspace

Best,

Yang

Niels Basjes <[hidden email]> 于2020年3月2日周一下午7:16写道：

Hi,

I'm running a lot of batch jobs on Kubernetes once in a while I get this exception.
What is causing this?
How can I fix this?

Niels Basjes

java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:45)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:53)
    at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:104)
    at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(ByteSizeValue.java:39)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:88)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:80)
    at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor.java:174)

--
Best regards / Met vriendelijke groeten,

Niels Basjes

Arvid Heise-3

Re: [Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

Hi Niels,

to add to Yang. 96m is plenty of space and was heavily tested by Alibaba.

The most likely reason and the motivation for the change is that you probably have a classloader leak in your pipeline, quite possibly by one of our connectors. For example, see FLINK-16142 [1].

If you could give us more details about your pipeline, we can try to ping it down.

However, if the error already occurs during the first start on the cluster, then the limit is too narrow and increasing it will already help. In that case, please leave us a quick comment with your used dependencies, so that we could potentially increase the default setting for future releases.

[1] https://issues.apache.org/jira/browse/FLINK-16142

On Mon, Mar 2, 2020 at 1:27 PM Yang Wang <[hidden email]> wrote:

From 1.10, Flink will enable the metaspace limit via "-XX:MaxMetaspaceSize"
by default. The default value is 96m, loading too many classes will cause
"OutOfMemoryError: Metaspace"[1]. So you need to increase the configured value.

[1]. https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_trouble.html#outofmemoryerror-metaspace

Best,
Yang

Niels Basjes <[hidden email]> 于2020年3月2日周一下午7:16写道：
Hi,

I'm running a lot of batch jobs on Kubernetes once in a while I get this exception.
What is causing this?
How can I fix this?

Niels Basjes

java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:45)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:53)
    at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:104)
    at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(ByteSizeValue.java:39)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:88)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:80)
    at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor.java:174)

--
Best regards / Met vriendelijke groeten,

Niels Basjes

Niels Basjes

Re: [Kubernetes] java.lang.OutOfMemoryError: Metaspace ?

I've put some information about my situation in the ticket

https://issues.apache.org/jira/browse/FLINK-16142?focusedCommentId=17049679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17049679

On Mon, Mar 2, 2020 at 2:55 PM Arvid Heise <[hidden email]> wrote:

Hi Niels,

to add to Yang. 96m is plenty of space and was heavily tested by Alibaba.

The most likely reason and the motivation for the change is that you probably have a classloader leak in your pipeline, quite possibly by one of our connectors. For example, see FLINK-16142 [1].
If you could give us more details about your pipeline, we can try to ping it down.

However, if the error already occurs during the first start on the cluster, then the limit is too narrow and increasing it will already help. In that case, please leave us a quick comment with your used dependencies, so that we could potentially increase the default setting for future releases.

[1] https://issues.apache.org/jira/browse/FLINK-16142

On Mon, Mar 2, 2020 at 1:27 PM Yang Wang <[hidden email]> wrote:
From 1.10, Flink will enable the metaspace limit via "-XX:MaxMetaspaceSize"
by default. The default value is 96m, loading too many classes will cause
"OutOfMemoryError: Metaspace"[1]. So you need to increase the configured value.

[1]. https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_trouble.html#outofmemoryerror-metaspace

Best,
Yang

Niels Basjes <[hidden email]> 于2020年3月2日周一下午7:16写道：
Hi,

I'm running a lot of batch jobs on Kubernetes once in a while I get this exception.
What is causing this?
How can I fix this?

Niels Basjes

java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:45)
    at org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:53)
    at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:104)
    at org.elasticsearch.common.unit.ByteSizeValue.<clinit>(ByteSizeValue.java:39)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:88)
    at org.elasticsearch.action.bulk.BulkProcessor$Builder.<init>(BulkProcessor.java:80)
    at org.elasticsearch.action.bulk.BulkProcessor.builder(BulkProcessor.java:174)

--
Best regards / Met vriendelijke groeten,

Niels Basjes

Best regards / Met vriendelijke groeten,

Niels Basjes