Flink exported metrics scope configuration

[Kai Fu]

Hi team,

We noticed that Prometheus metrics exporter exports all of the metrics at the most fine-grained level, which is tremendous for the prometheus server especially when the parallelism is high. The metrics volume crawled from a single host(parallelism 8) is around 40MB for us currently. This is due to task_name attribute in the metrics generated by the engine being very long. The task_name attribute is auto-generated from SQL job, and it seems it's attaching all field names onto it.

We want to reduce the metrics volume by either drop task_name or at some more coarse-grained level. But I cannot find any related documents about this, any advice on that? 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#list-of-all-variables

--

[Mason Chen]

Hi Kai,

You can use the excluded variables config for the reporter. 

• metrics.reporter.<name>.scope.variables.excludes: (optional) A semi-colon (;) separate list of variables that should be ignored by tag-based reporters (e.g., Prometheus, InfluxDB).

https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/#reporter

Best,

Mason

On Jun 3, 2021, at 9:31 PM, Kai Fu <[hidden email]> wrote:

Hi team,

We noticed that Prometheus metrics exporter exports all of the metrics at the most fine-grained level, which is tremendous for the prometheus server especially when the parallelism is high. The metrics volume crawled from a single host(parallelism 8) is around 40MB for us currently. This is due to task_name attribute in the metrics generated by the engine being very long. The task_name attribute is auto-generated from SQL job, and it seems it's attaching all field names onto it.

We want to reduce the metrics volume by either drop task_name or at some more coarse-grained level. But I cannot find any related documents about this, any advice on that? 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#list-of-all-variables

--

[Kai Fu]

Hi Mason,

Thank you for the advice, as I tried, it works and reduces a lot in size.

On Fri, Jun 4, 2021 at 11:45 AM Mason Chen <[hidden email]> wrote:

Hi Kai,

You can use the excluded variables config for the reporter. 

• metrics.reporter.<name>.scope.variables.excludes: (optional) A semi-colon (;) separate list of variables that should be ignored by tag-based reporters (e.g., Prometheus, InfluxDB).

https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/#reporter

Best,

Mason

On Jun 3, 2021, at 9:31 PM, Kai Fu <[hidden email]> wrote:

Hi team,

We noticed that Prometheus metrics exporter exports all of the metrics at the most fine-grained level, which is tremendous for the prometheus server especially when the parallelism is high. The metrics volume crawled from a single host(parallelism 8) is around 40MB for us currently. This is due to task_name attribute in the metrics generated by the engine being very long. The task_name attribute is auto-generated from SQL job, and it seems it's attaching all field names onto it.

We want to reduce the metrics volume by either drop task_name or at some more coarse-grained level. But I cannot find any related documents about this, any advice on that? 

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#list-of-all-variables

--

Best wishes,

- Kai

--