Does Flink automatically apply the rebalance operator ?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Does Flink automatically apply the rebalance operator ?

Suraj Puvvada
Hello

We are testing a simple use case where we read from kafka -> process and write to kafka.

I have set parallelism of the job to 3 and parallelism for the process function to 6. When I looked at the job graph in the Flink UI noticed that between the source and process function a rebalance step was introduced. Is this expected ?

I was expecting some instances of process tasks to be idle i.e. 3 tasks to be idle.

Thanks
Suraj
Reply | Threaded
Open this post in threaded view
|

Re: Does Flink automatically apply the rebalance operator ?

rmetzger0
Hi Suraj,

yes, this is the expected and desired behavior: a rebalance is introduced so that all 6 process functions are performing work.

On Thu, Jul 30, 2020 at 8:53 PM Suraj Puvvada <[hidden email]> wrote:
Hello

We are testing a simple use case where we read from kafka -> process and write to kafka.

I have set parallelism of the job to 3 and parallelism for the process function to 6. When I looked at the job graph in the Flink UI noticed that between the source and process function a rebalance step was introduced. Is this expected ?

I was expecting some instances of process tasks to be idle i.e. 3 tasks to be idle.

Thanks
Suraj