Direct Memory full

[Rex Fenley]

Hello,

Our job consistently shows

Outside JVM

Type	Count	Used	Capacity
Direct	32,839	1.03 GB	1.03 GB

for direct memory.

Is it typical for it to be full? What are the consequences that we may not be noticing of direct memory being full?

Thanks!

--

[rmetzger0]

Hey Rex,

the direct memory is used for IO. There is no concept of direct memory being "full". The only thing that can happen is that you have something in place (Kubernetes, YARN) that limits / enforces the memory use of a Flink process, and you run out of your memory allowance. The direct memory is allocated outside of the heap's upper limit, thus you could run out of the budget.

But Flink is usually properly configuring the memory limits correctly, to avoid running into those situations.

tl;dr: you don't need to worry about this.

On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <[hidden email]> wrote:

Hello,

Our job consistently shows

Outside JVM

Type	Count	Used	Capacity
Direct	32,839	1.03 GB	1.03 GB

for direct memory.

Is it typical for it to be full? What are the consequences that we may not be noticing of direct memory being full?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

[Rex Fenley]

Thanks for the reply. If what I'm understanding is correct there's no chance of an OOM, but since direct memory is for I/O, it being completely filled may be a sign of backpressure? Currently one of our operators takes a tremendous amount of time to align during a checkpoint. Could increasing direct memory help checkpointing by improving I/O performance across the whole plan (assuming I/O is at least part of the bottleneck)?

On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <[hidden email]> wrote:

Hey Rex,

the direct memory is used for IO. There is no concept of direct memory being "full". The only thing that can happen is that you have something in place (Kubernetes, YARN) that limits / enforces the memory use of a Flink process, and you run out of your memory allowance. The direct memory is allocated outside of the heap's upper limit, thus you could run out of the budget.

But Flink is usually properly configuring the memory limits correctly, to avoid running into those situations.

tl;dr: you don't need to worry about this.

On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <[hidden email]> wrote:

Hello,

Our job consistently shows

Outside JVM

Type	Count	Used	Capacity
Direct	32,839	1.03 GB	1.03 GB

for direct memory.

Is it typical for it to be full? What are the consequences that we may not be noticing of direct memory being full?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Steven Wu]

if you are running out of direct buffer, you will see "java.lang.OutOfMemoryError: Direct buffer memory"

On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <[hidden email]> wrote:

Thanks for the reply. If what I'm understanding is correct there's no chance of an OOM, but since direct memory is for I/O, it being completely filled may be a sign of backpressure? Currently one of our operators takes a tremendous amount of time to align during a checkpoint. Could increasing direct memory help checkpointing by improving I/O performance across the whole plan (assuming I/O is at least part of the bottleneck)?

On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <[hidden email]> wrote:

Hey Rex,

the direct memory is used for IO. There is no concept of direct memory being "full". The only thing that can happen is that you have something in place (Kubernetes, YARN) that limits / enforces the memory use of a Flink process, and you run out of your memory allowance. The direct memory is allocated outside of the heap's upper limit, thus you could run out of the budget.

But Flink is usually properly configuring the memory limits correctly, to avoid running into those situations.

tl;dr: you don't need to worry about this.

On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <[hidden email]> wrote:

Hello,

Our job consistently shows

Outside JVM

Type	Count	Used	Capacity
Direct	32,839	1.03 GB	1.03 GB

for direct memory.

Is it typical for it to be full? What are the consequences that we may not be noticing of direct memory being full?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

[rmetzger0]

I don't think the direct memory is causing any performance bottlenecks. The backpressure is probably caused by something else (high CPU load, slow external system, data skew)

On Wed, Dec 16, 2020 at 7:23 PM Steven Wu <[hidden email]> wrote:

if you are running out of direct buffer, you will see "java.lang.OutOfMemoryError: Direct buffer memory"

On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <[hidden email]> wrote:

Thanks for the reply. If what I'm understanding is correct there's no chance of an OOM, but since direct memory is for I/O, it being completely filled may be a sign of backpressure? Currently one of our operators takes a tremendous amount of time to align during a checkpoint. Could increasing direct memory help checkpointing by improving I/O performance across the whole plan (assuming I/O is at least part of the bottleneck)?

On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <[hidden email]> wrote:

Hey Rex,

the direct memory is used for IO. There is no concept of direct memory being "full". The only thing that can happen is that you have something in place (Kubernetes, YARN) that limits / enforces the memory use of a Flink process, and you run out of your memory allowance. The direct memory is allocated outside of the heap's upper limit, thus you could run out of the budget.

But Flink is usually properly configuring the memory limits correctly, to avoid running into those situations.

tl;dr: you don't need to worry about this.

On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <[hidden email]> wrote:

Hello,

Our job consistently shows

Outside JVM

Type	Count	Used	Capacity
Direct	32,839	1.03 GB	1.03 GB

for direct memory.

Is it typical for it to be full? What are the consequences that we may not be noticing of direct memory being full?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

[Rex Fenley]

Ok, thanks. I've said this in another thread but everything seems to go completely idle during checkpoints while waiting on 1 operator, there's no CPU usage, hardly any disk usage. I'll assume it's something else then.

On Wed, Dec 16, 2020 at 10:42 AM Robert Metzger <[hidden email]> wrote:

I don't think the direct memory is causing any performance bottlenecks. The backpressure is probably caused by something else (high CPU load, slow external system, data skew)

On Wed, Dec 16, 2020 at 7:23 PM Steven Wu <[hidden email]> wrote:

if you are running out of direct buffer, you will see "java.lang.OutOfMemoryError: Direct buffer memory"

On Wed, Dec 16, 2020 at 9:47 AM Rex Fenley <[hidden email]> wrote:

Thanks for the reply. If what I'm understanding is correct there's no chance of an OOM, but since direct memory is for I/O, it being completely filled may be a sign of backpressure? Currently one of our operators takes a tremendous amount of time to align during a checkpoint. Could increasing direct memory help checkpointing by improving I/O performance across the whole plan (assuming I/O is at least part of the bottleneck)?

On Tue, Dec 15, 2020 at 10:37 PM Robert Metzger <[hidden email]> wrote:

Hey Rex,

the direct memory is used for IO. There is no concept of direct memory being "full". The only thing that can happen is that you have something in place (Kubernetes, YARN) that limits / enforces the memory use of a Flink process, and you run out of your memory allowance. The direct memory is allocated outside of the heap's upper limit, thus you could run out of the budget.

But Flink is usually properly configuring the memory limits correctly, to avoid running into those situations.

tl;dr: you don't need to worry about this.

On Tue, Dec 15, 2020 at 8:38 AM Rex Fenley <[hidden email]> wrote:

Hello,

Our job consistently shows

Outside JVM

Type	Count	Used	Capacity
Direct	32,839	1.03 GB	1.03 GB

for direct memory.

Is it typical for it to be full? What are the consequences that we may not be noticing of direct memory being full?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--