Testing Flink with identity program in hibench

[penguin.]

Thanks for reading this email.

According to the introduction, the identity program in hibench reads data from Kafka and then writes it back to Kafka.

When using the identity program in hibench to test the Flink, set the parallelism to 20 in the flink.conf file in the conf directory of hibench.

After the task is submitted, only one subtask is found on the UI interface, that is, only one slot in the cluster is assigned a subtask. How can I have multiple subtasks when using identity to test Flink?

(it seems that every time the picture is uploaded, it can't be displayed, so no screenshot is provided.)

 

[Nicolaus Weidner]

Hi,

I am not familiar with hibench. Does the Flink UI show the configured parallelism of 20 for the job, and there are indeed 20 partitions on the Kafka topic you consume?

Which Flink version are you running? The repo https://github.com/Intel-bigdata/HiBench mentions Flink 1.0.3, which is very outdated. The current release is 1.13.0, so results may not be accurate.

Best wishes,

Nico

On Mon, May 17, 2021 at 10:28 AM penguin. <[hidden email]> wrote:

Thanks for reading this email.

According to the introduction, the identity program in hibench reads data from Kafka and then writes it back to Kafka.

When using the identity program in hibench to test the Flink, set the parallelism to 20 in the flink.conf file in the conf directory of hibench.

After the task is submitted, only one subtask is found on the UI interface, that is, only one slot in the cluster is assigned a subtask. How can I have multiple subtasks when using identity to test Flink?

(it seems that every time the picture is uploaded, it can't be displayed, so no screenshot is provided.)