As the answer on SO suggests, Prometheus comes with lots of functionality to do what you’re requesting using just a simple count metric:
https://prometheus.io/docs/prometheus/latest/querying/functions/
If you want to implement the function on your own inside flink, you can make your own metrics, and add them to the metric group, and use the metrics inside rich functions. For example:
getRuntimeContext().getMetricGroup().addGroup(YOUR_METRIC_GROUP_NAME)
This will return you a org.apache.flink.metrics.MetricGroup. You can then use this MetricGroup to attach Metrics, such as counters, gauges (in the SO suggestion), and histograms. You can then interact with these Metrics, and they will expose
your new metrics to your Prometheus exporter, for example.
Chris
From: aj <[hidden email]>
Date: Tuesday, June 16, 2020 at 1:31 PM
To: user <[hidden email]>
Subject: [EXTERNAL] Flink Count of Events using metric
Please help me with this:
I have a topic in Kafka where I am getting multiple types of events in JSON format. I have created a file stream sink to write these events to S3 with bucketing.
Now I want to publish an hourly count of each event as metrics to Prometheus and publish a grafana dashboard over that.
So please help me how can I achieve hourly count for each event using Flink metrics and publish to Prometheus.
--
Thanks & Regards,
Anuj Jain
Locked
