Integrating prometheus

[Manish G]

Hi,

I am following this link on how to integrate prometheus with flink.

Going by the code sample,  I would need to insert related metrics code in the main logic.

Is it avoidable, like by using some annotations on methods?

Manish

[rmetzger0]

Hi Manish,

Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach.

But of course, you can try to build your own metrics abstraction based on Flink's metric system.

On Fri, Jul 3, 2020 at 9:35 AM Manish G <[hidden email]> wrote:

Hi,

I am following this link on how to integrate prometheus with flink.

Going by the code sample,  I would need to insert related metrics code in the main logic.

Is it avoidable, like by using some annotations on methods?

Manish

[Chesnay Schepler]

What metrics specifically are you interested in?

On 03/07/2020 17:22, Robert Metzger wrote:

Hi Manish,

Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach.

But of course, you can try to build your own metrics abstraction based on Flink's metric system.

On Fri, Jul 3, 2020 at 9:35 AM Manish G <[hidden email]> wrote:

Hi,

I am following this link on how to integrate prometheus with flink.

Going by the code sample,  I would need to insert related metrics code in the main logic.

Is it avoidable, like by using some annotations on methods?

Manish

[Manish G]

Hi,

I am basically looking for : throughput, success rate, error rate.

For experimental purposes I could complete all configurations as explained in the official documentation. But somehow my custom metrics(a simple Counter) is still not shown on the prometheus board, though default metrics I can see.

Anything I am missing?

On Fri, Jul 3, 2020 at 8:57 PM Chesnay Schepler <[hidden email]> wrote:

What metrics specifically are you interested in?

On 03/07/2020 17:22, Robert Metzger wrote:

Hi Manish,

Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach.

But of course, you can try to build your own metrics abstraction based on Flink's metric system.

On Fri, Jul 3, 2020 at 9:35 AM Manish G <[hidden email]> wrote:

Hi,

I am following this link on how to integrate prometheus with flink.

Going by the code sample,  I would need to insert related metrics code in the main logic.

Is it avoidable, like by using some annotations on methods?

Manish

[Manish G]

Also, it seems custom metrics can only be recorded if we extend RichFunction, as it allows us to override open wherein we can get hold of context and metrics constructs.

Please let me know if there are other ways too.

On Fri, Jul 3, 2020 at 10:05 PM Manish G <[hidden email]> wrote:

Hi,

I am basically looking for : throughput, success rate, error rate.

For experimental purposes I could complete all configurations as explained in the official documentation. But somehow my custom metrics(a simple Counter) is still not shown on the prometheus board, though default metrics I can see.

Anything I am missing?

On Fri, Jul 3, 2020 at 8:57 PM Chesnay Schepler <[hidden email]> wrote:

What metrics specifically are you interested in?

On 03/07/2020 17:22, Robert Metzger wrote:

Hi Manish,

Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach.

But of course, you can try to build your own metrics abstraction based on Flink's metric system.

On Fri, Jul 3, 2020 at 9:35 AM Manish G <[hidden email]> wrote:

Hi,

I am following this link on how to integrate prometheus with flink.

Going by the code sample,  I would need to insert related metrics code in the main logic.

Is it avoidable, like by using some annotations on methods?

Manish

[Chesnay Schepler]

Yes, you do need to extend RichFunction; there's no way around that.

As for the missing metric, the usual cause is that the job/task finishes so quickly that the metric is never reported. If this is not the cause I would recommend enabling DEBUG logging and searching for warnings from the metric groups/registry/reporter.

On 03/07/2020 19:27, Manish G wrote:

Also, it seems custom metrics can only be recorded if we extend RichFunction, as it allows us to override open wherein we can get hold of context and metrics constructs.

Please let me know if there are other ways too.

On Fri, Jul 3, 2020 at 10:05 PM Manish G <[hidden email]> wrote:

Hi,

I am basically looking for : throughput, success rate, error rate.

For experimental purposes I could complete all configurations as explained in the official documentation. But somehow my custom metrics(a simple Counter) is still not shown on the prometheus board, though default metrics I can see.

Anything I am missing?

On Fri, Jul 3, 2020 at 8:57 PM Chesnay Schepler <[hidden email]> wrote:

What metrics specifically are you interested in?

On 03/07/2020 17:22, Robert Metzger wrote:

Hi Manish,

Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach.

But of course, you can try to build your own metrics abstraction based on Flink's metric system.

On Fri, Jul 3, 2020 at 9:35 AM Manish G <[hidden email]> wrote:

Hi,

I am following this link on how to integrate prometheus with flink.

Going by the code sample,  I would need to insert related metrics code in the main logic.

Is it avoidable, like by using some annotations on methods?

Manish