Client for Monitoring API!

[Anis Nasir]

Hey all,

Is there any client library that we can use to fetch/store the metrics expose through flink monitoring rest api ?

Regards,

Anis

[Felipe Gutierrez]

yes.

you can use prometheus+Grafana.

https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter

https://felipeogutierrez.blogspot.com/2019/04/monitoring-apache-flink-with-prometheus.html

Felipe

On 2019/09/18 11:36:37, Anis Nasir <[hidden email]> wrote:
> Hey all,>
>
> Is there any client library that we can use to fetch/store the metrics>
> expose through flink monitoring rest api ?>
>
>
> Regards,>
> Anis>
>

[Biao Liu]

Hi Anis,

Have you tried Flink metric reporter? It's a better way to handle metrics than through rest api.

Flink supports reporting metrics to external system. You could find the list of external systems supported here [1].

1. https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/metrics.html#reporter

On Wed, 18 Sep 2019 at 19:36, Anis Nasir <[hidden email]> wrote:

Hey all,

Is there any client library that we can use to fetch/store the metrics expose through flink monitoring rest api ?

Regards,

Anis

[Anis Nasir]

Thanks Biao for your response.

We would like to fetch metrics at subtask level for each checkpoint. This information is not exposed via default metrics, but are available in rest end point!

Also, would like to persist the history of checkpoints. This information is lost whenever we restart the job (or job manager is lost).

Cheers,

Anis

 

On Thu, Sep 19, 2019 at 5:02 AM Biao Liu <[hidden email]> wrote:

Hi Anis,

Have you tried Flink metric reporter? It's a better way to handle metrics than through rest api.

Flink supports reporting metrics to external system. You could find the list of external systems supported here [1].

1. https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/metrics.html#reporter

On Wed, 18 Sep 2019 at 19:36, Anis Nasir <[hidden email]> wrote:

Hey all,

Is there any client library that we can use to fetch/store the metrics expose through flink monitoring rest api ?

Regards,

Anis

[Biao Liu]

Ah, now I understand what exactly your requirement is.

I don't think there is such a tool in Flink which could help you to fetch and store the content of rest api. It seems not to be a general requirement.

But I'm really interested in the motivation behind your requirement. Could you share more about it?

Why do you want to keep the history of checkpointing at subtask level? How do you use this history?

On Thu, 19 Sep 2019 at 16:12, Anis Nasir <[hidden email]> wrote:

Thanks Biao for your response.

We would like to fetch metrics at subtask level for each checkpoint. This information is not exposed via default metrics, but are available in rest end point!

Also, would like to persist the history of checkpoints. This information is lost whenever we restart the job (or job manager is lost).

Cheers,

Anis

 

On Thu, Sep 19, 2019 at 5:02 AM Biao Liu <[hidden email]> wrote:

Hi Anis,

Have you tried Flink metric reporter? It's a better way to handle metrics than through rest api.

Flink supports reporting metrics to external system. You could find the list of external systems supported here [1].

1. https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/metrics.html#reporter

Thanks,

Biao /'bɪ.aʊ/

On Wed, 18 Sep 2019 at 19:36, Anis Nasir <[hidden email]> wrote:

Hey all,

Is there any client library that we can use to fetch/store the metrics expose through flink monitoring rest api ?

Regards,

Anis