Change in sub-task id assignment from 1.9 to 1.10?

[Ken Krugler]

Hi all,

Was there any change in how sub-tasks get allocated to TMs, from Flink 1.9 to 1.10?

Specifically for consecutively numbered sub-tasks (e.g. 0, 1, 2) did it become more or less likely that they’d be allocated to the same Task Manager?

Asking because a workflow that ran fine in 1.9 now has a “hot” TM that’s having trouble keeping up with a Kafka topic.

The most promising explanation is that now there are three sub-tasks on the same TM that are reading from that topic, versus previously they’d be scattered across multiple TMs.

But I don’t see significant changes in this area post 1.8

Thanks,

— Ken

--------------------------

Ken Krugler

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[Zhu Zhu]

Hi Ken,

There were no such changes in my mind.

And in Flink there was no designed logic to scatter subtasks of the same 

operator into different taskmanagers. 

One workaround to solve your problem could be to increase the parallelism of 

your source vertex to be no smaller than no other operator so that each 

slot can contain a source task. With config cluster.evenly-spread-out-slots 

set to true, slots can be evenly distributed in all available taskmanagers in most cases.

Thanks,

Zhu Zhu

Ken Krugler <[hidden email]> 于2020年8月7日周五 上午5:28写道：

Hi all,

Was there any change in how sub-tasks get allocated to TMs, from Flink 1.9 to 1.10?

Specifically for consecutively numbered sub-tasks (e.g. 0, 1, 2) did it become more or less likely that they’d be allocated to the same Task Manager?

Asking because a workflow that ran fine in 1.9 now has a “hot” TM that’s having trouble keeping up with a Kafka topic.

The most promising explanation is that now there are three sub-tasks on the same TM that are reading from that topic, versus previously they’d be scattered across multiple TMs.

But I don’t see significant changes in this area post 1.8

Thanks,

— Ken

--------------------------

Ken Krugler

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr