Way to get accumulators values *during* job execution ?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Way to get accumulators values *during* job execution ?

LINZ, Arnaud

Hello,

 

I use Grafana/Graphite to monitor my applications. The Flink GUI is really nice, but it disappears after the job completes and consequently is not suitable to long-term monitoring.

 

For batch applications, I simply send the accumulator’s values at the end of the job to my Graphite base.

For streaming applications, it’s more complex as the job never ends. It would be nice to have a way of getting current accumulator values (like in the GUI)  to push it periodically to Graphite in a monitoring thread. Is there any API to get the values during execution ?

 

Best regards,

Arnaud




L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
Reply | Threaded
Open this post in threaded view
|

Re: Way to get accumulators values *during* job execution ?

Stephan Ewen
Hi Arnaud!

One thing you can do is to periodically retrieve them by querying the monitoring API: https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html

A nice approach would be to let the JobManager eagerly publish the metrics.  I think that Christian Kreizfeld was working on an extension that periodically publishes the accumulators to some metrics store. Maybe he can comment on that.

Greetings,
Stephan


On Wed, Dec 2, 2015 at 5:21 PM, LINZ, Arnaud <[hidden email]> wrote:

Hello,

 

I use Grafana/Graphite to monitor my applications. The Flink GUI is really nice, but it disappears after the job completes and consequently is not suitable to long-term monitoring.

 

For batch applications, I simply send the accumulator’s values at the end of the job to my Graphite base.

For streaming applications, it’s more complex as the job never ends. It would be nice to have a way of getting current accumulator values (like in the GUI)  to push it periodically to Graphite in a monitoring thread. Is there any API to get the values during execution ?

 

Best regards,

Arnaud




L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.