Capturing statistics for more than 5 minutes

[Nomchin Banga]

Hi

We are a group of graduate students from Purdue University who are doing an experimental study to compare different data ingestion engines.

For this purpose, we are trying to collect statistics of running jobs over a few days. However, Flink’s UI captures the statistics for the last 5 minutes.
We tried to hit the Rest API but that does not contain the data of statistics other than bytes read/written.

It would be very helpful if you could guide us regarding a way in which we can capture the entire history of metrics.

Looking forward to hearing from you

Regards

Nomchin Banga

[Chesnay Schepler]

Hello,

the Metric System should be exactly what you're looking for.

On 15.11.2017 03:55, Nomchin Banga wrote:

Hi

We are a group of graduate students from Purdue University who are doing an experimental study to compare different data ingestion engines.

For this purpose, we are trying to collect statistics of running jobs over a few days. However, Flink’s UI captures the statistics for the last 5 minutes.
We tried to hit the Rest API but that does not contain the data of statistics other than bytes read/written.

It would be very helpful if you could guide us regarding a way in which we can capture the entire history of metrics.

Looking forward to hearing from you

Regards

Nomchin Banga

[Nomchin Banga]

Hi

I have been able to run JMX with Flink with the following configuration applied to the flink-conf.yaml file of all nodes in the cluster:

metrics.reporters: jmx

metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter

metrics.reporter.jmx.port: 9020-9022

env.java.opts: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

When I run JConsole and listen on ports master-IP:9999/slave-IP:9020, I am able to see the system metrics like CPU, memory etc.

How can I access the task metrics and their respective graphs like bytesRead, latency etc. which are collected for each subtask and shown on the GUI.

Any help will be appreciated!

Thanks

On Wed, Nov 15, 2017 at 1:12 AM, Chesnay Schepler <[hidden email]> wrote:

Hello,

the Metric System should be exactly what you're looking for.

On 15.11.2017 03:55, Nomchin Banga wrote:

Hi

We are a group of graduate students from Purdue University who are doing an experimental study to compare different data ingestion engines.

For this purpose, we are trying to collect statistics of running jobs over a few days. However, Flink’s UI captures the statistics for the last 5 minutes.
We tried to hit the Rest API but that does not contain the data of statistics other than bytes read/written.

It would be very helpful if you could guide us regarding a way in which we can capture the entire history of metrics.

Looking forward to hearing from you

Regards

Nomchin Banga

--