Operator latency metric not working in 1.9.1

[orips]

We've been working with Flink 1.5.2 with Prometheus and ever since we upgraded to version 1.9.1 the operator latency metric is not available anymore (flink_taskmanager_job_latency_source_id_source_subtask_index_operator_id_operator_subtask_index_latency)

I've seen in the docs that this metric is still supported. Does anyone know why it happens?

Thanks

[Rafi Aroch]

Hi Ori,

Make sure that latency metrics is enabled. It's disabled by default. See also that the scope is set properly.

https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#latency-tracking

https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#metrics-latency-interval

Thanks, 

Rafi

On Sun, Mar 1, 2020, 13:32 Ori Popowski <[hidden email]> wrote:

We've been working with Flink 1.5.2 with Prometheus and ever since we upgraded to version 1.9.1 the operator latency metric is not available anymore (flink_taskmanager_job_latency_source_id_source_subtask_index_operator_id_operator_subtask_index_latency)

I've seen in the docs that this metric is still supported. Does anyone know why it happens?

Thanks

[orips]

Thanks for the response.

In 1.5 the docs also state that it should be enabled [1], however, it always
worked without setting latencyTrackingInterval

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#latency-tracking



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Gary Yao-5]

Hi,

There is a release note for Flink 1.7 that could be relevant for you [1]

    Granularity of latency metrics
    The default granularity for latency metrics has been modified. To restore the previous behavior users have to explicitly set the granularity to subtask.

Best,
Gary

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/release-notes/flink-1.7.html#granularity-of-latency-metrics

On Tue, Mar 3, 2020 at 10:14 AM orips <[hidden email]> wrote:

Thanks for the response.

In 1.5 the docs also state that it should be enabled [1], however, it always
worked without setting latencyTrackingInterval

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#latency-tracking



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[orips]

Thanks, that makes sense!

In addition, I've just found the reason for this in the code:

This is 1.5 (default value is 2000L):
https://github.com/apache/flink/blob/release-1.5/flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java#L109

This is 1.9 (default value is 0L)
https://github.com/apache/flink/blob/release-1.9/flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java#L115




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/