Unable to override metric format for Prometheus Reporter

[Kaustubh Rudrawar]

Hi,

I'm setting up Flink 1.7.0 on a Kubernetes cluster and am seeing some unexpected behavior when using the Prometheus Reporter.

With the following setup in flink-conf.yaml: 

    metrics.reporters: prometheus

    metrics.reporter.prometheus.class: org.apache.flink.metrics.prometheus.PrometheusReporter

   

I am able to make a request to the job manager at the default port (9249) and retrieve metrics that look like this:

# HELP flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time Time (scope: jobmanager_Status_JVM_GarbageCollector_Copy)

# TYPE flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time gauge

flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time{host="localhost",} 0.0

# HELP flink_jobmanager_taskSlotsTotal taskSlotsTotal (scope: jobmanager)

# TYPE flink_jobmanager_taskSlotsTotal gauge

flink_jobmanager_taskSlotsTotal{host="localhost",} 0.0

What I would like to do is change the format of these metrics using the scope overrides. I've defined the following overrides, in flink-conf.yaml, as a test and they are not taking effect. Is there some other configuration I need to have in place for these overrides to work?

    metrics.scope.jm: flink.testcluster.jobmanager.dev.<host>

    metrics.scope.jm.job: flink.testjob.jobmanager.<job_name>

When I put these overrides in, the job manager logs from org.apache.flink.configuration.GlobalConfiguration show that my overrides are being seen by Flink. 

Thanks for your help! 

-Kaustubh

[Chesnay Schepler]

The prometheus reporter ignores scope formats. In fact all reporters that work with tags (i.e., key-value-pairs) ignore them, the idea being that you would search specific metrics based on their tags.
I'm not aware of any intermediate workarounds.

There is a JIRA for this issue: https://issues.apache.org/jira/browse/FLINK-9898

On 15.01.2019 20:45, Kaustubh Rudrawar wrote:

Hi,

I'm setting up Flink 1.7.0 on a Kubernetes cluster and am seeing some unexpected behavior when using the Prometheus Reporter.

With the following setup in flink-conf.yaml: 

    metrics.reporters: prometheus

    metrics.reporter.prometheus.class: org.apache.flink.metrics.prometheus.PrometheusReporter

   

I am able to make a request to the job manager at the default port (9249) and retrieve metrics that look like this:

# HELP flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time Time (scope: jobmanager_Status_JVM_GarbageCollector_Copy)

# TYPE flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time gauge

flink_jobmanager_Status_JVM_GarbageCollector_Copy_Time{host="localhost",} 0.0

# HELP flink_jobmanager_taskSlotsTotal taskSlotsTotal (scope: jobmanager)

# TYPE flink_jobmanager_taskSlotsTotal gauge

flink_jobmanager_taskSlotsTotal{host="localhost",} 0.0

What I would like to do is change the format of these metrics using the scope overrides. I've defined the following overrides, in flink-conf.yaml, as a test and they are not taking effect. Is there some other configuration I need to have in place for these overrides to work?

    metrics.scope.jm: flink.testcluster.jobmanager.dev.<host>

    metrics.scope.jm.job: flink.testjob.jobmanager.<job_name>

When I put these overrides in, the job manager logs from org.apache.flink.configuration.GlobalConfiguration show that my overrides are being seen by Flink. 

Thanks for your help! 

-Kaustubh