How do I get the value of 99th latency inside an operator?

[Felipe Gutierrez]

Hi community,

where from the Dlink code I can get the value of 99th percentile latency (flink_taskmanager_job_latency_source_id_operator_id_operator_subtask_index_latency{operator_id="93352199ce18d8917f20fdf82cedb1b4",quantile="0.99"})?

Probably I will have to hack the Flink source code to export those values to my own operator. Nevertheless, it is what I need.

Kind Regards,

Felipe

[Aljoscha Krettek]

Hi,

I'm afraid you're correct, this is currently not exposed and you would
have to hack around some things and/or use reflection.

AbstractStreamOperator has a field latencyStats [1], which is what holds
the metrics. This is being updated from method
reportOrUpdateLatencyMarker [2].

I hope that helps somewhat.

Best,
Aljoscha

[1]
https://github.com/apache/flink/blob/ab642cb616fb909893e2c650b0b4c2aa10407e6d/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L154
[2]
https://github.com/apache/flink/blob/ab642cb616fb909893e2c650b0b4c2aa10407e6d/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L702

On 05.03.20 18:17, Felipe Gutierrez wrote:

> Hi community,
>
> where from the Dlink code I can get the value of 99th percentile latency
> (flink_taskmanager_job_latency_source_id_operator_id_operator_subtask_index_latency{operator_id="93352199ce18d8917f20fdf82cedb1b4",quantile="0.99"})?
>
> Probably I will have to hack the Flink source code to export those values
> to my own operator. Nevertheless, it is what I need.
>
> Kind Regards,
> Felipe
>
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>

[Arvid Heise-3]

Hi Felipe,

could you use the JMX metrics reporter and tap into the reported values?

The proposed hacks are obviously unstable over time.

On Fri, Mar 6, 2020 at 1:06 PM Aljoscha Krettek <[hidden email]> wrote:

Hi,

I'm afraid you're correct, this is currently not exposed and you would
have to hack around some things and/or use reflection.

AbstractStreamOperator has a field latencyStats [1], which is what holds
the metrics. This is being updated from method
reportOrUpdateLatencyMarker [2].

I hope that helps somewhat.

Best,
Aljoscha

[1]
https://github.com/apache/flink/blob/ab642cb616fb909893e2c650b0b4c2aa10407e6d/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L154
[2]
https://github.com/apache/flink/blob/ab642cb616fb909893e2c650b0b4c2aa10407e6d/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L702

On 05.03.20 18:17, Felipe Gutierrez wrote:
> Hi community,
>
> where from the Dlink code I can get the value of 99th percentile latency
> (flink_taskmanager_job_latency_source_id_operator_id_operator_subtask_index_latency{operator_id="93352199ce18d8917f20fdf82cedb1b4",quantile="0.99"})?
>
> Probably I will have to hack the Flink source code to export those values
> to my own operator. Nevertheless, it is what I need.
>
> Kind Regards,
> Felipe
>
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>