(DEPRECATED) Apache Flink User Mailing List archive.

How to get throughput and processing latency when testing Flink with hibench?

Classic

List

Threaded

2 messages Options

penguin.

How to get throughput and processing latency when testing Flink with hibench?

When using hibench to test Flink, the jobs submitted are built-in applications of hibench, that is, the code logic of programs like wordcount cannot be changed.

How can I get the throughput and processing delay of Flink?

In addition, in the /report/hibench.report file of hibench, we can't get the throughput information of Flink. The contents of /report/hibench.report are as follows:

Type	Date	Time	Duration(s)
FlinkFixWindow	2021-05-17	16:17:58	88.599
FlinkFixWindow	2021-05-17	16:33:12	872.982
FlinkFixWindow	2021-05-17	16:53:29	301.524
FlinkFixWindow	2021-05-17	17:09:22	880.098

Nicolaus Weidner

Re: How to get throughput and processing latency when testing Flink with hibench?

Hi,

Flink allows to enable latency tracking [1] and exposes several metrics that might be what you are looking for [2, look for e.g. "numRecordsIn" or "numBytesIn"]. You can query these metrics using the REST API [3] or by registering a metrics reporter [4] that exposes them. As noted in the other email, I cannot speak about what hibench does and why it cannot show any throughput information for Flink. Even Flink 1.0.3 seems to expose some metrics (just going by the docs [5] here, I was not involved back then).

Best wishes,

Nico

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#latency-tracking

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#io

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/rest_api/

[4] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/

[5] https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/monitoring_rest_api.html

On Mon, May 17, 2021 at 2:49 PM penguin. <[hidden email]> wrote:

When using hibench to test Flink, the jobs submitted are built-in applications of hibench, that is, the code logic of programs like wordcount cannot be changed.
How can I get the throughput and processing delay of Flink?
In addition, in the /report/hibench.report file of hibench, we can't get the throughput information of Flink. The contents of /report/hibench.report are as follows:
Type Date Time Input_data_size Duration(s) Throunghput(bytes/s) Throunghput/node
FlinkFixWindow 2021-05-17 16:17:58 0 88.599 0 0
FlinkFixWindow 2021-05-17 16:33:12 0 872.982 0 0
FlinkFixWindow 2021-05-17 16:53:29 0 301.524 0 0
FlinkFixWindow 2021-05-17 17:09:22 0 880.098 0 0