What is "Processed (persisted) in-flight data"

[Rex Fenley]

Hello,

In the latest version I now see two byte measures on checkpoints. There's

Checkpointed Data Size

9.05 GB

Which I'm assuming means 9.05 GB were written in this incremental checkpoint.

But now there is also

Processed (persisted) in-flight data

152 MB (0 B)

What is that?

Thanks!

--

[Arvid Heise-4]

Processed in-flight data is the size of data that is processed between the first and last checkpoint barrier in aligned checkpointing. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html#history-tab

On Sun, Jan 31, 2021 at 7:45 AM Rex Fenley <[hidden email]> wrote:

Hello,

In the latest version I now see two byte measures on checkpoints. There's

Checkpointed Data Size

9.05 GB

Which I'm assuming means 9.05 GB were written in this incremental checkpoint.

But now there is also

Processed (persisted) in-flight data

152 MB (0 B)

What is that?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

[Rex Fenley]

Got it, thanks! What is the 0 B part of that?

On Sun, Jan 31, 2021 at 3:43 AM Arvid Heise <[hidden email]> wrote:

Processed in-flight data is the size of data that is processed between the first and last checkpoint barrier in aligned checkpointing. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html#history-tab

On Sun, Jan 31, 2021 at 7:45 AM Rex Fenley <[hidden email]> wrote:

Hello,

In the latest version I now see two byte measures on checkpoints. There's

Checkpointed Data Size

9.05 GB

Which I'm assuming means 9.05 GB were written in this incremental checkpoint.

But now there is also

Processed (persisted) in-flight data

152 MB (0 B)

What is that?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Rex Fenley]

Oh, nvm, that's the "Persisted" part which is documented as "Persisted in-flight data: The number of bytes persisted during the alignment (time between receiving the first and the last checkpoint barrier) over all acknowledged subtasks. This is > 0 only if the unaligned checkpoints are enabled."

Thanks!

On Sun, Jan 31, 2021 at 3:10 PM Rex Fenley <[hidden email]> wrote:

Got it, thanks! What is the 0 B part of that?

On Sun, Jan 31, 2021 at 3:43 AM Arvid Heise <[hidden email]> wrote:

Processed in-flight data is the size of data that is processed between the first and last checkpoint barrier in aligned checkpointing. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html#history-tab

On Sun, Jan 31, 2021 at 7:45 AM Rex Fenley <[hidden email]> wrote:

Hello,

In the latest version I now see two byte measures on checkpoints. There's

Checkpointed Data Size

9.05 GB

Which I'm assuming means 9.05 GB were written in this incremental checkpoint.

But now there is also

Processed (persisted) in-flight data

152 MB (0 B)

What is that?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--