Job tasks are not balance among taskmanagers

[Ever]

Hi, there're 3 taskManager nodes within our testing flink cluster, whose version is 1.8. And each one have 10 taskslots.

Now I have a job with parallelism 3. 

I expected the 3 tasks will be located at 3 different taskManagers, just as Example 2 below:

But it came out that all 3 tasks are all located at the same taskmanager.

Why?

[Haibo Sun]

Hi, Ever

This is a regression wrt the pre Flip-6 code, and the following JIRA dedicated to this issue.

https://issues.apache.org/jira/browse/FLINK-12122 

 

Best,

Haibo

At 2019-07-02 11:42:27, "Ever" <[hidden email]> wrote:

Hi, there're 3 taskManager nodes within our testing flink cluster, whose version is 1.8. And each one have 10 taskslots.

Now I have a job with parallelism 3. 

I expected the 3 tasks will be located at 3 different taskManagers, just as Example 2 below:

But it came out that all 3 tasks are all located at the same taskmanager.

Why?

[Ken Krugler]

Hi Ever,

As Haibo noted, that’s a known regression.

If you fall back to the older approach of having multiple TMs per slave, each with one slot, then Flink (as of 1.7/1.8) does a better job of distributing work.

— Ken

On Jul 1, 2019, at 9:23 PM, Haibo Sun <[hidden email]> wrote:

Hi, Ever

This is a regression wrt the pre Flip-6 code, and the following JIRA dedicated to this issue.

https://issues.apache.org/jira/browse/FLINK-12122 

 

Best,

Haibo

At 2019-07-02 11:42:27, "Ever" <[hidden email]> wrote:

Hi, there're 3 taskManager nodes within our testing flink cluster, whose version is 1.8. And each one have 10 taskslots.

Now I have a job with parallelism 3. 

I expected the 3 tasks will be located at 3 different taskManagers, just as Example 2 below:

<[hidden email]>

But it came out that all 3 tasks are all located at the same taskmanager.

<[hidden email]>

<[hidden email]>

Why?

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra