Hi, I need to capture the Metrics sent from a Flink app to a Reporter and transform them to an Events API format I have designed. I have been looking at the Reporters(https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables) and have used them but what would be a best practice to capture this metrics data to transform it ? The folks using the Flink app still want to see their metrics in the Flink Dashboard using their chosen(not sure yet what they chose-assuming ConsoleReporter) Reporter. I need to capture those metrics, transform them to my Events API format and send it to a Kinesis Stream. We use Prometheus and InfluxDB in our environments for other purposes. Should I use the SLF4J Reporter to dump the metrics into a log file/folder and watch that with a Kinesis Agent and transform it somehow(?) and then send it to the Kinesis data stream ? TIA, |
Hi, Vijay Or can you implement a Reporter that transforms the metrics and sends them directly to a Kinesis Stream? Best, Haibo At 2019-07-16 00:01:36, "Vijay Balakrishnan" <[hidden email]> wrote:
|
You can configure multiple reporters,
so just configured both the reporter that the app users want and
your own which does the transformation and sending to kinesis.
On 16/07/2019 09:37, Haibo Sun wrote:
|
Thanks for all your replies.Ended up using a StatsdReporter with Flink and building a statsd plugin to transform the data to my required output format and dump it into a folder that the Kinesis agent can then pick up. On Tue, Jul 16, 2019 at 2:16 AM Chesnay Schepler <[hidden email]> wrote:
|
Free forum by Nabble | Edit this page |