Export user metrics with Flink Prometheus endpoint

[Eleanore Jin]

Hi all, 

I just wonder is it possible to use Flink Metrics endpoint to allow Prometheus to scrape user defined metrics? 

Context: 

In addition to Flink metrics, we also collect some application level metrics using opencensus. And we run opencensus agent as side car in kubernetes pod to collect metrics (opencensus agent talks to task manager container via rpcs) 

The issue with this approach is: it looks like opencensus agent keeps staled metrics, causing the metrics reporting inaccurate, and this project is not actively maintained anymore. 

So I wonder if it is possible to use Flink metrics endpoint for user defined metrics. 

Thanks a lot! 

Eleanore 

[Aljoscha Krettek]

Hi,

that should be possible. Did you have a look at the documentation for
setting up a Prometheus metrics reporter:
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter

Best,
Aljoscha

On 06.05.20 03:02, Eleanore Jin wrote:

> Hi all,
>
> I just wonder is it possible to use Flink Metrics endpoint to allow
> Prometheus to scrape user defined metrics?
>
> Context:
> In addition to Flink metrics, we also collect some application level
> metrics using opencensus. And we run opencensus agent as side car in
> kubernetes pod to collect metrics (opencensus agent talks to task manager
> container via rpcs)
>
> The issue with this approach is: it looks like opencensus agent keeps
> staled metrics, causing the metrics reporting inaccurate, and this project
> is not actively maintained anymore.
>
> So I wonder if it is possible to use Flink metrics endpoint for user
> defined metrics.
>
> Thanks a lot!
> Eleanore
>

[Eleanore Jin]

Hi Aljoscha,

Thanks for the response, yes prometheus reporter is already enabled and I am able to get the flink metrics from prometheus. 

My question is more like, currently I am using opencensus library to collect application metrics, so is there an easy way to integrate this metrics with flink metrics endpoint. 

Thanks!

Eleanore

On Wed, May 6, 2020 at 7:48 AM Aljoscha Krettek <[hidden email]> wrote:

Hi,

that should be possible. Did you have a look at the documentation for
setting up a Prometheus metrics reporter:
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter

Best,
Aljoscha

On 06.05.20 03:02, Eleanore Jin wrote:
> Hi all,
>
> I just wonder is it possible to use Flink Metrics endpoint to allow
> Prometheus to scrape user defined metrics?
>
> Context:
> In addition to Flink metrics, we also collect some application level
> metrics using opencensus. And we run opencensus agent as side car in
> kubernetes pod to collect metrics (opencensus agent talks to task manager
> container via rpcs)
>
> The issue with this approach is: it looks like opencensus agent keeps
> staled metrics, causing the metrics reporting inaccurate, and this project
> is not actively maintained anymore.
>
> So I wonder if it is possible to use Flink metrics endpoint for user
> defined metrics.
>
> Thanks a lot!
> Eleanore
>

[Aljoscha Krettek]

Now I see what you mean. I think you would have to somehow set up the
Flink metrics system as a backend for opencensus. Then the metrics would
be reported to the same system (prometheus) in this case. In Opencensus
lingo, this would mean using a Flink-based Stats Exporter instead of the
Prometheus backend directly.

I don't have experience developing opencensus Stats Exporters but
working off the provided ones should work. Maybe have a look at the
prometheus exporter [1] and go from there. The entrypoint for a Flink
exporter would then probably be

     FlinkStatsCollector.createAndRegister(getMetricGroup());

Best,
Aljoscha

On 06.05.20 17:11, Eleanore Jin wrote:

> Hi Aljoscha,
>
> Thanks for the response, yes prometheus reporter is already enabled and I
> am able to get the flink metrics from prometheus.
>
> My question is more like, currently I am using opencensus library
> <https://github.com/census-instrumentation/opencensus-java> to collect
> application metrics, so is there an easy way to integrate this metrics with
> flink metrics endpoint.
>
> Thanks!
> Eleanore
>
> On Wed, May 6, 2020 at 7:48 AM Aljoscha Krettek <[hidden email]> wrote:
>
>> Hi,
>>
>> that should be possible. Did you have a look at the documentation for
>> setting up a Prometheus metrics reporter:
>>
>> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter
>>
>> Best,
>> Aljoscha
>>
>> On 06.05.20 03:02, Eleanore Jin wrote:
>>> Hi all,
>>>
>>> I just wonder is it possible to use Flink Metrics endpoint to allow
>>> Prometheus to scrape user defined metrics?
>>>
>>> Context:
>>> In addition to Flink metrics, we also collect some application level
>>> metrics using opencensus. And we run opencensus agent as side car in
>>> kubernetes pod to collect metrics (opencensus agent talks to task manager
>>> container via rpcs)
>>>
>>> The issue with this approach is: it looks like opencensus agent keeps
>>> staled metrics, causing the metrics reporting inaccurate, and this
>> project
>>> is not actively maintained anymore.
>>>
>>> So I wonder if it is possible to use Flink metrics endpoint for user
>>> defined metrics.
>>>
>>> Thanks a lot!
>>> Eleanore
>>>
>>
>>
>

[Eleanore Jin]

Hi Aljoscha, 

Thanks a lot for the suggestion!

Best,

Eleanore

On Mon, May 18, 2020 at 2:16 AM Aljoscha Krettek <[hidden email]> wrote:

Now I see what you mean. I think you would have to somehow set up the
Flink metrics system as a backend for opencensus. Then the metrics would
be reported to the same system (prometheus) in this case. In Opencensus
lingo, this would mean using a Flink-based Stats Exporter instead of the
Prometheus backend directly.

I don't have experience developing opencensus Stats Exporters but
working off the provided ones should work. Maybe have a look at the
prometheus exporter [1] and go from there. The entrypoint for a Flink
exporter would then probably be

     FlinkStatsCollector.createAndRegister(getMetricGroup());

Best,
Aljoscha

On 06.05.20 17:11, Eleanore Jin wrote:
> Hi Aljoscha,
>
> Thanks for the response, yes prometheus reporter is already enabled and I
> am able to get the flink metrics from prometheus.
>
> My question is more like, currently I am using opencensus library
> <https://github.com/census-instrumentation/opencensus-java> to collect
> application metrics, so is there an easy way to integrate this metrics with
> flink metrics endpoint.
>
> Thanks!
> Eleanore
>
> On Wed, May 6, 2020 at 7:48 AM Aljoscha Krettek <[hidden email]> wrote:
>
>> Hi,
>>
>> that should be possible. Did you have a look at the documentation for
>> setting up a Prometheus metrics reporter:
>>
>> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter
>>
>> Best,
>> Aljoscha
>>
>> On 06.05.20 03:02, Eleanore Jin wrote:
>>> Hi all,
>>>
>>> I just wonder is it possible to use Flink Metrics endpoint to allow
>>> Prometheus to scrape user defined metrics?
>>>
>>> Context:
>>> In addition to Flink metrics, we also collect some application level
>>> metrics using opencensus. And we run opencensus agent as side car in
>>> kubernetes pod to collect metrics (opencensus agent talks to task manager
>>> container via rpcs)
>>>
>>> The issue with this approach is: it looks like opencensus agent keeps
>>> staled metrics, causing the metrics reporting inaccurate, and this
>> project
>>> is not actively maintained anymore.
>>>
>>> So I wonder if it is possible to use Flink metrics endpoint for user
>>> defined metrics.
>>>
>>> Thanks a lot!
>>> Eleanore
>>>
>>
>>
>