Renaming the metrics

[orips]

I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark).

I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name).

How can I do it?

Thanks.

[Arvid Heise-3]

Hi Ori,

I see that the PrometheusPushGatewayReporter [1] has an option for a job name, maybe you can use that.

I'm also including Chesnay who probably has more ideas.

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter

On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <[hidden email]> wrote:

I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark).

I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name).

How can I do it?

Thanks.

--

[Chesnay Schepler]

There's currently no way to change this.

A related enhancement was proposed on FLINK-17495 that would at least allow you to attach a custom label, but the initial implementation wasn't general enough.

On 22/06/2020 15:08, Arvid Heise wrote:

Hi Ori,

I see that the PrometheusPushGatewayReporter [1] has an option for a job name, maybe you can use that.

I'm also including Chesnay who probably has more ideas.

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter

On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <[hidden email]> wrote:

I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark).

I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name).

How can I do it?

Thanks.

--

[orips]

Thanks for answering.

Unrelated to Flink, but if anyone knows a way to rename the metrics inside Prometheus I'd appreciate if you can share.

About the push gateway - I think I'll stick with the pull options because it looks like a better fit to the use case

On Mon, Jun 22, 2020 at 4:47 PM Chesnay Schepler <[hidden email]> wrote:

There's currently no way to change this.

A related enhancement was proposed on FLINK-17495 that would at least allow you to attach a custom label, but the initial implementation wasn't general enough.

On 22/06/2020 15:08, Arvid Heise wrote:

Hi Ori,

I see that the PrometheusPushGatewayReporter [1] has an option for a job name, maybe you can use that.

I'm also including Chesnay who probably has more ideas.

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter

On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <[hidden email]> wrote:

I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark).

I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name).

How can I do it?

Thanks.

--

Arvid Heise | Senior Java Developer

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng   

[cslotterback]

Hi Ori,

 

Another more temporary brute-force option, while not officially flink, could be building a modified version of the metrics plugin into flink where you manually manipulate the prefixes yourself. It’s actually pretty easy to build the jar, and to test it you drop the jar into the plugin path. I’ve done something similar where I actually filter our a lot of the prefixes that I don’t want, because too many metric points were being generated from some custom metrics. The config for the filter is loaded from the flink conf, you could possibly implement something similar where you pass the job name in each clusters config:

 

https://github.com/cslotterback/flink/commit/fd8e1f77a83a3ae1253da53596d22471bb6fe902

and

https://github.com/cslotterback/flink/commit/ce3797ea46f3321885c4352ecc36b9385b7ca0ce

 

This isn’t what I’d call ideal, but it gets the job done. I would love a generic flink-approved method of configuring Prometheus metrics.

 

Chris

 

 

From: Ori Popowski <[hidden email]>
Date: Monday, June 22, 2020 at 12:22 PM
Cc: user <[hidden email]>
Subject: [EXTERNAL] Re: Renaming the metrics

 

Thanks for answering.

 

Unrelated to Flink, but if anyone knows a way to rename the metrics inside Prometheus I'd appreciate if you can share.

 

About the push gateway - I think I'll stick with the pull options because it looks like a better fit to the use case

 

On Mon, Jun 22, 2020 at 4:47 PM Chesnay Schepler <[hidden email]> wrote:

There's currently no way to change this.

 

A related enhancement was proposed on FLINK-17495 that would at least allow you to attach a custom label, but the initial implementation wasn't general enough.

 

On 22/06/2020 15:08, Arvid Heise wrote:

Hi Ori,

 

I see that the PrometheusPushGatewayReporter [1] has an option for a job name, maybe you can use that.

 

I'm also including Chesnay who probably has more ideas.

 

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter

 

On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <[hidden email]> wrote:

I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark).

 

I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name).

 

How can I do it?

 

Thanks.

--

Arvid Heise | Senior Java Developer

 

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng   

 

[orips]

Thanks for the suggestion.

After digging a bit, we've found it most convenient to just add labels to all our Prometheus queries, like this:

flink_taskmanager_job_task_operator_currentOutputWatermark{job_name="<job-name>"}

The job_name label will be exposed if you run your job with a job name like this:

senv.execute("<job-name>")

On Mon, Jun 22, 2020 at 8:01 PM Slotterback, Chris <[hidden email]> wrote:

Hi Ori,

 

Another more temporary brute-force option, while not officially flink, could be building a modified version of the metrics plugin into flink where you manually manipulate the prefixes yourself. It’s actually pretty easy to build the jar, and to test it you drop the jar into the plugin path. I’ve done something similar where I actually filter our a lot of the prefixes that I don’t want, because too many metric points were being generated from some custom metrics. The config for the filter is loaded from the flink conf, you could possibly implement something similar where you pass the job name in each clusters config:

 

https://github.com/cslotterback/flink/commit/fd8e1f77a83a3ae1253da53596d22471bb6fe902

and

https://github.com/cslotterback/flink/commit/ce3797ea46f3321885c4352ecc36b9385b7ca0ce

 

This isn’t what I’d call ideal, but it gets the job done. I would love a generic flink-approved method of configuring Prometheus metrics.

 

Chris

 

 

From: Ori Popowski <[hidden email]>
Date: Monday, June 22, 2020 at 12:22 PM
Cc: user <[hidden email]>
Subject: [EXTERNAL] Re: Renaming the metrics

 

Thanks for answering.

 

Unrelated to Flink, but if anyone knows a way to rename the metrics inside Prometheus I'd appreciate if you can share.

 

About the push gateway - I think I'll stick with the pull options because it looks like a better fit to the use case

 

On Mon, Jun 22, 2020 at 4:47 PM Chesnay Schepler <[hidden email]> wrote:

There's currently no way to change this.

 

A related enhancement was proposed on FLINK-17495 that would at least allow you to attach a custom label, but the initial implementation wasn't general enough.

 

On 22/06/2020 15:08, Arvid Heise wrote:

Hi Ori,

 

I see that the PrometheusPushGatewayReporter [1] has an option for a job name, maybe you can use that.

 

I'm also including Chesnay who probably has more ideas.

 

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter

 

On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <[hidden email]> wrote:

I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark).

 

I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name).

 

How can I do it?

 

Thanks.

--

Arvid Heise | Senior Java Developer

 

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng