(DEPRECATED) Apache Flink User Mailing List archive.

inputFloatingBuffersUsage=0?

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
Alexey Trenikhun
Reply | Threaded
Open this post in threaded view
♦
♦
|

inputFloatingBuffersUsage=0?

Alexey Trenikhun
Hello,
While trying to investigate back pressure using hints from [1], I've noticed that flink_taskmanager_job_task_buffers_inPoolUsage and flink_taskmanager_job_task_buffers_inputFloatingBuffersUsage are always 0, which looks suspicious, are these metrics still populated ?

Thanks,
Alexey
Piotr Nowojski-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: inputFloatingBuffersUsage=0?

Piotr Nowojski-4
Hi Alexey,

Have you looked at the documentation [1]?

> inPoolUsage An estimate of the input buffers usage. (ignores LocalInputChannels) Gauge
> inputFloatingBuffersUsage An estimate of the floating input buffers usage. (ignores LocalInputChannels) Gauge
> inputExclusiveBuffersUsage An estimate of the exclusive input buffers usage. (ignores LocalInputChannels) Gauge

Second reason might be simply that your job/task didn't manage to acquire all of the recommended memory. Recommended memory is `floatingBuffers + exclusiveBuffers * numberOfChannels`. Required memory is just `1 * numberOfChannels`. If there is not enough memory to satisfy recommended values, Flink will work with anything between recommended and required amounts.

Best,
Piotrek


czw., 18 mar 2021 o 08:30 Alexey Trenikhun <[hidden email]> napisał(a):
Hello,
While trying to investigate back pressure using hints from [1], I've noticed that flink_taskmanager_job_task_buffers_inPoolUsage and flink_taskmanager_job_task_buffers_inputFloatingBuffersUsage are always 0, which looks suspicious, are these metrics still populated ?

Thanks,
Alexey

Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing. 23 Jul 2019 Nico Kruber & Piotr Nowojski . In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details.This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify ...

Piotr Nowojski-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: inputFloatingBuffersUsage=0?

Piotr Nowojski-4
Hi,

clarification about the 2nd part. Required memory is one single exclusive buffer per channel, so if you are running low on memory, floating buffers are one of the first to go, hence you could observe some `0` for floating buffers.

For analysing backpressure I would recommend to use a different metric: `isBackPressured`. It was added in Flink 1.10, after this blog post had been written. Since 1.11 we also have `idleTimeMsPerSecond` that might be helpful. 

In 1.13 all of this will become even more simple, with changes to Web UI and an addition of `backPressuredTimeMsPerSecond` and `busyTimeMsPerSecond`. We are planning to release a new updated blog post about analysing backpressure in the following weeks.

Best,
Piotrek

pt., 19 mar 2021 o 11:57 Piotr Nowojski <[hidden email]> napisał(a):
Hi Alexey,

Have you looked at the documentation [1]?

> inPoolUsage An estimate of the input buffers usage. (ignores LocalInputChannels) Gauge
> inputFloatingBuffersUsage An estimate of the floating input buffers usage. (ignores LocalInputChannels) Gauge
> inputExclusiveBuffersUsage An estimate of the exclusive input buffers usage. (ignores LocalInputChannels) Gauge

Second reason might be simply that your job/task didn't manage to acquire all of the recommended memory. Recommended memory is `floatingBuffers + exclusiveBuffers * numberOfChannels`. Required memory is just `1 * numberOfChannels`. If there is not enough memory to satisfy recommended values, Flink will work with anything between recommended and required amounts.

Best,
Piotrek


czw., 18 mar 2021 o 08:30 Alexey Trenikhun <[hidden email]> napisał(a):
Hello,
While trying to investigate back pressure using hints from [1], I've noticed that flink_taskmanager_job_task_buffers_inPoolUsage and flink_taskmanager_job_task_buffers_inputFloatingBuffersUsage are always 0, which looks suspicious, are these metrics still populated ?

Thanks,
Alexey

Apache Flink: Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing
Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing. 23 Jul 2019 Nico Kruber & Piotr Nowojski . In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details.This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify ...
flink.apache.org

Alexey Trenikhun
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: inputFloatingBuffersUsage=0?

Alexey Trenikhun
Hi Piotrek,

Thank for information, looks like isBackPressured ( and in future backPressuredTimeMsPerSecond) is more useful for our simple monitoring purposes. Looking forward for updated blog post

Thanks,
Alexey

From: Piotr Nowojski <[hidden email]>
Sent: Friday, March 19, 2021 5:01 AM
To: Alexey Trenikhun <[hidden email]>
Cc: Flink User Mail List <[hidden email]>
Subject: Re: inputFloatingBuffersUsage=0?
 
Hi,

clarification about the 2nd part. Required memory is one single exclusive buffer per channel, so if you are running low on memory, floating buffers are one of the first to go, hence you could observe some `0` for floating buffers.

For analysing backpressure I would recommend to use a different metric: `isBackPressured`. It was added in Flink 1.10, after this blog post had been written. Since 1.11 we also have `idleTimeMsPerSecond` that might be helpful. 

In 1.13 all of this will become even more simple, with changes to Web UI and an addition of `backPressuredTimeMsPerSecond` and `busyTimeMsPerSecond`. We are planning to release a new updated blog post about analysing backpressure in the following weeks.

Best,
Piotrek

pt., 19 mar 2021 o 11:57 Piotr Nowojski <[hidden email]> napisał(a):
Hi Alexey,

Have you looked at the documentation [1]?

> inPoolUsage An estimate of the input buffers usage. (ignores LocalInputChannels) Gauge
> inputFloatingBuffersUsage An estimate of the floating input buffers usage. (ignores LocalInputChannels) Gauge
> inputExclusiveBuffersUsage An estimate of the exclusive input buffers usage. (ignores LocalInputChannels) Gauge

Second reason might be simply that your job/task didn't manage to acquire all of the recommended memory. Recommended memory is `floatingBuffers + exclusiveBuffers * numberOfChannels`. Required memory is just `1 * numberOfChannels`. If there is not enough memory to satisfy recommended values, Flink will work with anything between recommended and required amounts.

Best,
Piotrek


czw., 18 mar 2021 o 08:30 Alexey Trenikhun <[hidden email]> napisał(a):
Hello,
While trying to investigate back pressure using hints from [1], I've noticed that flink_taskmanager_job_task_buffers_inPoolUsage and flink_taskmanager_job_task_buffers_inputFloatingBuffersUsage are always 0, which looks suspicious, are these metrics still populated ?

Thanks,
Alexey

Apache Flink: Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing
Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing. 23 Jul 2019 Nico Kruber & Piotr Nowojski . In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details.This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify ...
flink.apache.org

Piotr Nowojski-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: inputFloatingBuffersUsage=0?

Piotr Nowojski-4
Great, thanks for letting me know. 

pt., 19 mar 2021 o 20:24 Alexey Trenikhun <[hidden email]> napisał(a):
Hi Piotrek,

Thank for information, looks like isBackPressured ( and in future backPressuredTimeMsPerSecond) is more useful for our simple monitoring purposes. Looking forward for updated blog post

Thanks,
Alexey

From: Piotr Nowojski <[hidden email]>
Sent: Friday, March 19, 2021 5:01 AM
To: Alexey Trenikhun <[hidden email]>
Cc: Flink User Mail List <[hidden email]>
Subject: Re: inputFloatingBuffersUsage=0?
 
Hi,

clarification about the 2nd part. Required memory is one single exclusive buffer per channel, so if you are running low on memory, floating buffers are one of the first to go, hence you could observe some `0` for floating buffers.

For analysing backpressure I would recommend to use a different metric: `isBackPressured`. It was added in Flink 1.10, after this blog post had been written. Since 1.11 we also have `idleTimeMsPerSecond` that might be helpful. 

In 1.13 all of this will become even more simple, with changes to Web UI and an addition of `backPressuredTimeMsPerSecond` and `busyTimeMsPerSecond`. We are planning to release a new updated blog post about analysing backpressure in the following weeks.

Best,
Piotrek

pt., 19 mar 2021 o 11:57 Piotr Nowojski <[hidden email]> napisał(a):
Hi Alexey,

Have you looked at the documentation [1]?

> inPoolUsage An estimate of the input buffers usage. (ignores LocalInputChannels) Gauge
> inputFloatingBuffersUsage An estimate of the floating input buffers usage. (ignores LocalInputChannels) Gauge
> inputExclusiveBuffersUsage An estimate of the exclusive input buffers usage. (ignores LocalInputChannels) Gauge

Second reason might be simply that your job/task didn't manage to acquire all of the recommended memory. Recommended memory is `floatingBuffers + exclusiveBuffers * numberOfChannels`. Required memory is just `1 * numberOfChannels`. If there is not enough memory to satisfy recommended values, Flink will work with anything between recommended and required amounts.

Best,
Piotrek


czw., 18 mar 2021 o 08:30 Alexey Trenikhun <[hidden email]> napisał(a):
Hello,
While trying to investigate back pressure using hints from [1], I've noticed that flink_taskmanager_job_task_buffers_inPoolUsage and flink_taskmanager_job_task_buffers_inputFloatingBuffersUsage are always 0, which looks suspicious, are these metrics still populated ?

Thanks,
Alexey

Apache Flink: Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing
Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing. 23 Jul 2019 Nico Kruber & Piotr Nowojski . In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details.This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify ...
flink.apache.org

Free forum by Nabble Edit this page