Programmatically get live values of accumulators

[Gwenhael Pasquiers]

Hi, and best wishes for the year to come J

 

I’d like to be able to programmatically get the (live) values of accumulators in order to send them using a statsd (or another) client in the JobManager of a yarn-deployed application. I say live because I’d like to use that in streaming (24/7) applications, and send live stats, I cannot way for the application to end.

 

I’ve seen that there is a json API (I’d prefer no to have my app poll itself).

I’ve seen some code on github (tests files) where it’s done using the underlying akka framework, I don’t mind doing it the same way and creating an actor to get notifications messages, but I don’t know the best way, and there probably is a better one.

 

Thanks in advance,

 

Gwenhaël PASQUIERS

[Jamie Grier]

Hi Gwenhael,

I think what you actually want is to use the Apache Flink metrics interface.  See the following: https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html

Sending metrics to StatsD is supported out-of-the-box.

-Jamie

On Mon, Jan 2, 2017 at 1:34 AM, Gwenhael Pasquiers <[hidden email]> wrote:

Hi, and best wishes for the year to come J

 

I’d like to be able to programmatically get the (live) values of accumulators in order to send them using a statsd (or another) client in the JobManager of a yarn-deployed application. I say live because I’d like to use that in streaming (24/7) applications, and send live stats, I cannot way for the application to end.

 

I’ve seen that there is a json API (I’d prefer no to have my app poll itself).

I’ve seen some code on github (tests files) where it’s done using the underlying akka framework, I don’t mind doing it the same way and creating an actor to get notifications messages, but I don’t know the best way, and there probably is a better one.

 

Thanks in advance,

 

Gwenhaël PASQUIERS

--

[Gwenhael Pasquiers]

Indeed, that looks like what we need. We currently rely on flink 1.0.1, that feature might be a good reason to update our flink version. I’ll test it. Many thanks ! !

 

From: Jamie Grier [mailto:[hidden email]]
Sent: lundi 2 janvier 2017 20:56
To: [hidden email]
Subject: Re: Programmatically get live values of accumulators

 

Hi Gwenhael,

 

I think what you actually want is to use the Apache Flink metrics interface.  See the following: https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html

 

Sending metrics to StatsD is supported out-of-the-box.

 

-Jamie

 

 

On Mon, Jan 2, 2017 at 1:34 AM, Gwenhael Pasquiers <[hidden email]> wrote:

Hi, and best wishes for the year to come J

 

I’d like to be able to programmatically get the (live) values of accumulators in order to send them using a statsd (or another) client in the JobManager of a yarn-deployed application. I say live because I’d like to use that in streaming (24/7) applications, and send live stats, I cannot way for the application to end.

 

I’ve seen that there is a json API (I’d prefer no to have my app poll itself).

I’ve seen some code on github (tests files) where it’s done using the underlying akka framework, I don’t mind doing it the same way and creating an actor to get notifications messages, but I don’t know the best way, and there probably is a better one.

 

Thanks in advance,

 

Gwenhaël PASQUIERS

 

--

 

Jamie Grier

data Artisans, Director of Applications Engineering

@jamiegrier

[hidden email]