Apache Flink - Flink Metrics collection using Prometheus on EMR from streaming mode

[M Singh]

Hi:

I wanted to find out what's the best way of collecting Flink metrics using Prometheus in a streaming application on EMR/Hadoop.

Since the Flink streaming jobs could be running on any node - is there any Prometheus configuration or service discovery option available that will dynamically pick up the metrics from the Filnk job and task managers running in cluster ?  

I believe for a batch job I can configure flink config to use Prometheus gateway configuration but I think this is not recommended for a streaming job.

Please let me know if you have any advice.

Thanks

Mans

[vino yang]

Hi Mans,

IMO, the mechanism of metrics reporter does not depend on any deployment mode.

>> is there any Prometheus configuration or service discovery option available that will dynamically pick up the metrics from the Filnk job and task managers running in cluster ?

Can you share more information about your scene?

>> I believe for a batch job I can configure flink config to use Prometheus gateway configuration but I think this is not recommended for a streaming job.

What does this mean? Why the Prometheus gateway configuration for Flink batch job is not recommended for a streaming job？

Best,

Vino

M Singh <[hidden email]> 于2019年12月24日周二 下午4:02写道：

Hi:

I wanted to find out what's the best way of collecting Flink metrics using Prometheus in a streaming application on EMR/Hadoop.

Since the Flink streaming jobs could be running on any node - is there any Prometheus configuration or service discovery option available that will dynamically pick up the metrics from the Filnk job and task managers running in cluster ?  

I believe for a batch job I can configure flink config to use Prometheus gateway configuration but I think this is not recommended for a streaming job.

Please let me know if you have any advice.

Thanks

Mans

[Rafi Aroch]

Hi,

Take a look here: https://github.com/eastcirclek/flink-service-discovery

I used it successfully quite a while ago, so things might have changed since.

Thanks, 

Rafi 

On Wed, Dec 25, 2019, 05:54 vino yang <[hidden email]> wrote:

Hi Mans,

IMO, the mechanism of metrics reporter does not depend on any deployment mode.

>> is there any Prometheus configuration or service discovery option available that will dynamically pick up the metrics from the Filnk job and task managers running in cluster ?

Can you share more information about your scene?

>> I believe for a batch job I can configure flink config to use Prometheus gateway configuration but I think this is not recommended for a streaming job.

What does this mean? Why the Prometheus gateway configuration for Flink batch job is not recommended for a streaming job？

Best,

Vino

M Singh <[hidden email]> 于2019年12月24日周二 下午4:02写道：

Hi:

I wanted to find out what's the best way of collecting Flink metrics using Prometheus in a streaming application on EMR/Hadoop.

Since the Flink streaming jobs could be running on any node - is there any Prometheus configuration or service discovery option available that will dynamically pick up the metrics from the Filnk job and task managers running in cluster ?  

I believe for a batch job I can configure flink config to use Prometheus gateway configuration but I think this is not recommended for a streaming job.

Please let me know if you have any advice.

Thanks

Mans

[M Singh]

Thanks Vino and Rafi for your references.

Regarding push gateway recommendations for batch - I am following this reference (https://prometheus.io/docs/practices/pushing/).

The scenario that I have is that we start Flink Apps on EMR whenever we need them. Sometimes the task manager gets killed and then restarted on another node.  In order to keep up with registering new task/job managers and de-registering the stopped/removed ones, I wanted to see if there is any service discovery integration with Flink apps.  

Thanks again for your help and let me know if you have any additional pointers.

On Wednesday, December 25, 2019, 03:39:31 AM EST, Rafi Aroch <[hidden email]> wrote:

Hi,

Take a look here: https://github.com/eastcirclek/flink-service-discovery

I used it successfully quite a while ago, so things might have changed since.

Thanks, 

Rafi 

On Wed, Dec 25, 2019, 05:54 vino yang <[hidden email]> wrote:

Hi Mans,

IMO, the mechanism of metrics reporter does not depend on any deployment mode.

>> is there any Prometheus configuration or service discovery option available that will dynamically pick up the metrics from the Filnk job and task managers running in cluster ?

Can you share more information about your scene?

>> I believe for a batch job I can configure flink config to use Prometheus gateway configuration but I think this is not recommended for a streaming job.

What does this mean? Why the Prometheus gateway configuration for Flink batch job is not recommended for a streaming job？

Best,

Vino

M Singh <[hidden email]> 于2019年12月24日周二 下午4:02写道：

Hi:

I wanted to find out what's the best way of collecting Flink metrics using Prometheus in a streaming application on EMR/Hadoop.

Since the Flink streaming jobs could be running on any node - is there any Prometheus configuration or service discovery option available that will dynamically pick up the metrics from the Filnk job and task managers running in cluster ?  

I believe for a batch job I can configure flink config to use Prometheus gateway configuration but I think this is not recommended for a streaming job.

Please let me know if you have any advice.

Thanks

Mans