Converting Metrics from a Reporter to a Custom Events mapping

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Converting Metrics from a Reporter to a Custom Events mapping

Vijay Balakrishnan
Hi,
I need to capture the Metrics sent from a Flink app to a Reporter and transform them to an Events API format I have designed. I have been looking at the Reporters(https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables) and have used them but what would be a best practice to capture this metrics data to transform it ?

The folks using the Flink app still want to see their metrics in the Flink Dashboard using their chosen(not sure yet what they chose-assuming ConsoleReporter) Reporter. I need to capture those metrics, transform them to my Events API format and send it to a Kinesis Stream.

We use Prometheus and InfluxDB in our environments for other purposes.

Should I use the SLF4J Reporter to dump the metrics into a log file/folder and watch that with a Kinesis Agent and transform it somehow(?) and then send it to the Kinesis data stream ?

TIA,
Reply | Threaded
Open this post in threaded view
|

Re:Converting Metrics from a Reporter to a Custom Events mapping

Haibo Sun
Hi,  Vijay

Or can you implement a Reporter that transforms the metrics and sends them directly to a Kinesis Stream?

Best,
Haibo

At 2019-07-16 00:01:36, "Vijay Balakrishnan" <[hidden email]> wrote:
Hi,
I need to capture the Metrics sent from a Flink app to a Reporter and transform them to an Events API format I have designed. I have been looking at the Reporters(https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables) and have used them but what would be a best practice to capture this metrics data to transform it ?

The folks using the Flink app still want to see their metrics in the Flink Dashboard using their chosen(not sure yet what they chose-assuming ConsoleReporter) Reporter. I need to capture those metrics, transform them to my Events API format and send it to a Kinesis Stream.

We use Prometheus and InfluxDB in our environments for other purposes.

Should I use the SLF4J Reporter to dump the metrics into a log file/folder and watch that with a Kinesis Agent and transform it somehow(?) and then send it to the Kinesis data stream ?

TIA,
Reply | Threaded
Open this post in threaded view
|

Re: Converting Metrics from a Reporter to a Custom Events mapping

Chesnay Schepler
You can configure multiple reporters, so just configured both the reporter that the app users want and your own which does the transformation and sending to kinesis.

On 16/07/2019 09:37, Haibo Sun wrote:
Hi,  Vijay

Or can you implement a Reporter that transforms the metrics and sends them directly to a Kinesis Stream?

Best,
Haibo

At 2019-07-16 00:01:36, "Vijay Balakrishnan" [hidden email] wrote:
Hi,
I need to capture the Metrics sent from a Flink app to a Reporter and transform them to an Events API format I have designed. I have been looking at the Reporters(https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables) and have used them but what would be a best practice to capture this metrics data to transform it ?

The folks using the Flink app still want to see their metrics in the Flink Dashboard using their chosen(not sure yet what they chose-assuming ConsoleReporter) Reporter. I need to capture those metrics, transform them to my Events API format and send it to a Kinesis Stream.

We use Prometheus and InfluxDB in our environments for other purposes.

Should I use the SLF4J Reporter to dump the metrics into a log file/folder and watch that with a Kinesis Agent and transform it somehow(?) and then send it to the Kinesis data stream ?

TIA,


Reply | Threaded
Open this post in threaded view
|

Re: Converting Metrics from a Reporter to a Custom Events mapping

Vijay Balakrishnan
Thanks for all your replies.Ended up using a StatsdReporter with Flink and building a statsd plugin to transform the data to my required output format and dump it into a folder that the Kinesis agent can then pick up.

On Tue, Jul 16, 2019 at 2:16 AM Chesnay Schepler <[hidden email]> wrote:
You can configure multiple reporters, so just configured both the reporter that the app users want and your own which does the transformation and sending to kinesis.

On 16/07/2019 09:37, Haibo Sun wrote:
Hi,  Vijay

Or can you implement a Reporter that transforms the metrics and sends them directly to a Kinesis Stream?

Best,
Haibo

At 2019-07-16 00:01:36, "Vijay Balakrishnan" [hidden email] wrote:
Hi,
I need to capture the Metrics sent from a Flink app to a Reporter and transform them to an Events API format I have designed. I have been looking at the Reporters(https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables) and have used them but what would be a best practice to capture this metrics data to transform it ?

The folks using the Flink app still want to see their metrics in the Flink Dashboard using their chosen(not sure yet what they chose-assuming ConsoleReporter) Reporter. I need to capture those metrics, transform them to my Events API format and send it to a Kinesis Stream.

We use Prometheus and InfluxDB in our environments for other purposes.

Should I use the SLF4J Reporter to dump the metrics into a log file/folder and watch that with a Kinesis Agent and transform it somehow(?) and then send it to the Kinesis data stream ?

TIA,