Re: understadning kafka connector - rebalance

Posted by avilevi on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/understadning-kafka-connector-rebalance-tp24724p24734.html

Ok, thanks for the clarification. but if I use it with keyed state so the partition is by the key. rebalancing will not shuffle this partitioning ? e.g 

.addSource(source)

      .rebalance

      .keyBy(_.id)

      .mapWithState(...)

On Mon, Nov 26, 2018 at 8:32 AM Taher Koitawala <[hidden email]> wrote:

Hi Avi,

          No, rebalance is not changing the number of kafka partitions. Lets say you have 6 kafka partitions and your flink parallelism is 8, in this case using rebalance will send records to all downstream operators in a round robin fashion. 

On Mon, Nov 26, 2018 at 11:33 AM Avi Levi <[hidden email]> wrote:

Hi

Looking at this example, doing the "rebalance" (e.g messageStream.rebalance().map(...) ) operation on heavy load stream wouldn't slow the stream ? is the rebalancing action occurs only when there is a partition change ? 

it says that "the rebelance call is causing a repartitioning of the data so that all machines" is it actually changing the num of partitions of the topic to match the num of flink operators ?

Avi