Metrics for number of "open windows"?

[Andrew Roberts]

Hello,

I’m trying to track the number of currently-in-state windows in a keyed, windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using Flink metrics. Are there any built in? Or any good approaches for collecting this data?

Thanks,

Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any

attachments may be confidential. If you are not an intended recipient, you

are hereby notified that any dissemination, distribution or copying of this

e-mail is strictly prohibited. If you have received this e-mail in error,

please notify the sender and permanently delete the e-mail and any

attachments immediately. You should not retain, copy or use this e-mail or

any attachment for any purpose, nor disclose all or any part of the

contents to any other person. Thank you.*

[Rong Rong]

Hi Andrew,

I am assuming you are actually using customized windowAssigner, trigger and process function. 

I think the best way for you to keep in-flight, not-yet-triggered windows is to emit metrics in these 3 pieces. 

Upon looking at the window operator, I don't think there's a a metrics (guage) that keeps how many windows are not yet fired. 

This information is available in the KeyedStateBackend, but I dont think the KeyedStateBackend is emitting any metrics related to what you want.

Thanks,

Rong

On Tue, Feb 19, 2019 at 12:14 PM Andrew Roberts <[hidden email]> wrote:

Hello,

I’m trying to track the number of currently-in-state windows in a keyed, windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using Flink metrics. Are there any built in? Or any good approaches for collecting this data?

Thanks,

Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any

attachments may be confidential. If you are not an intended recipient, you

are hereby notified that any dissemination, distribution or copying of this

e-mail is strictly prohibited. If you have received this e-mail in error,

please notify the sender and permanently delete the e-mail and any

attachments immediately. You should not retain, copy or use this e-mail or

any attachment for any purpose, nor disclose all or any part of the

contents to any other person. Thank you.*

[Andrey Zagrebin-3]

Hi Andrew,

Just to add the Rong's answer, if you use RocksDB state backend, you can activate state metrics forwarded from RocksDB [1].

Best,

Andrey

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#rocksdb

On Thu, Feb 21, 2019 at 11:22 PM Rong Rong <[hidden email]> wrote:

Hi Andrew,

I am assuming you are actually using customized windowAssigner, trigger and process function. 

I think the best way for you to keep in-flight, not-yet-triggered windows is to emit metrics in these 3 pieces. 

Upon looking at the window operator, I don't think there's a a metrics (guage) that keeps how many windows are not yet fired. 

This information is available in the KeyedStateBackend, but I dont think the KeyedStateBackend is emitting any metrics related to what you want.

Thanks,

Rong

On Tue, Feb 19, 2019 at 12:14 PM Andrew Roberts <[hidden email]> wrote:

Hello,

I’m trying to track the number of currently-in-state windows in a keyed, windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using Flink metrics. Are there any built in? Or any good approaches for collecting this data?

Thanks,

Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any

attachments may be confidential. If you are not an intended recipient, you

are hereby notified that any dissemination, distribution or copying of this

e-mail is strictly prohibited. If you have received this e-mail in error,

please notify the sender and permanently delete the e-mail and any

attachments immediately. You should not retain, copy or use this e-mail or

any attachment for any purpose, nor disclose all or any part of the

contents to any other person. Thank you.*

[Konstantin Knauf-2]

Hi Andrew,

when you implement your own Trigger or customize an existing Trigger you get access to the TriggerContext in all callbacks of the Trigger interface. The TriggerContext allows you to register custom metrics via TriggerContext:getMetricGroup() and  you can use partitioned state scoped to key and window of the current element via the TriggerContext:getPartitionedState(). It should be possible to use those two ingredients to add a custom metric with the number of open windows.

Cheers,

Konstantin

On Mon, Feb 25, 2019 at 10:58 AM Andrey Zagrebin <[hidden email]> wrote:

Hi Andrew,

Just to add the Rong's answer, if you use RocksDB state backend, you can activate state metrics forwarded from RocksDB [1].

Best,

Andrey

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#rocksdb

On Thu, Feb 21, 2019 at 11:22 PM Rong Rong <[hidden email]> wrote:

Hi Andrew,

I am assuming you are actually using customized windowAssigner, trigger and process function. 

I think the best way for you to keep in-flight, not-yet-triggered windows is to emit metrics in these 3 pieces. 

Upon looking at the window operator, I don't think there's a a metrics (guage) that keeps how many windows are not yet fired. 

This information is available in the KeyedStateBackend, but I dont think the KeyedStateBackend is emitting any metrics related to what you want.

Thanks,

Rong

On Tue, Feb 19, 2019 at 12:14 PM Andrew Roberts <[hidden email]> wrote:

Hello,

I’m trying to track the number of currently-in-state windows in a keyed, windowed stream (stream.keyBy(…).window(…).trigger(…).process(…)) using Flink metrics. Are there any built in? Or any good approaches for collecting this data?

Thanks,

Andrew
--
*Confidentiality Notice: The information contained in this e-mail and any

attachments may be confidential. If you are not an intended recipient, you

are hereby notified that any dissemination, distribution or copying of this

e-mail is strictly prohibited. If you have received this e-mail in error,

please notify the sender and permanently delete the e-mail and any

attachments immediately. You should not retain, copy or use this e-mail or

any attachment for any purpose, nor disclose all or any part of the

contents to any other person. Thank you.*

--