Flink 1.7.2: All jobs are getting deployed on the same task manager

[HarshithBolar]

Hi all,

 

We're running a Flink on a five node standalone cluster with three task manager (TM1, TM2, TM3) and two job managers.

 

Whenever I submit a new job, the job gets deployed on only TM3. When the number of slots in TM3 get exhausted, the jobs start getting deployed on TM2 and so on. How do I ensure that the jobs get distributed evenly across all 3 task managers?

 

Thanks,

Harshith

 

[Andrea Spina]

Hi everybody. We're currently experimenting the same characteristic on flink-1.6.2.

I've been reading that Flink treats all the slot as equals, it doesn't even know where these slots reside https://stackoverflow.com/questions/54980104/uneven-assignment-of-tasks-to-workers-in-flink.
So it should not be an issue; thus, the fact that it runs all the slots of a machine before moving to a new one should be just a rough coincidence.

Given that, I'm pretty sure that I've never been recording this feature using previous majors (I recall flink-1.3 for sure).
Moreover, this is damaging because you can get resources exhausted (e.g. memory, disk).

Hope we might find a solution on this.
Sincerely, 

Andrea

Il giorno lun 18 mar 2019 alle ore 11:53 Kumar Bolar, Harshith <[hidden email]> ha scritto:

Hi all,

 

We're running a Flink on a five node standalone cluster with three task manager (TM1, TM2, TM3) and two job managers.

 

Whenever I submit a new job, the job gets deployed on only TM3. When the number of slots in TM3 get exhausted, the jobs start getting deployed on TM2 and so on. How do I ensure that the jobs get distributed evenly across all 3 task managers?

 

Thanks,

Harshith

 

--

[HarshithBolar]

We recently upgraded to 1.7.2. Previously in 1.4.2 the jobs were getting deployed in a round robin manner. First job goes to TM1, second job goes to TM2 and so on. But after upgrading to 1.7.2, all jobs are going to the same task manager until the slots there run out. Did something change in 1.7?

 

Thanks,

Harshith

 

From: Andrea Spina <[hidden email]>
Date: Monday, 18 March 2019 at 5:54 PM
To: Harshith Kumar Bolar <[hidden email]>
Cc: user <[hidden email]>
Subject: [External] Re: Flink 1.7.2: All jobs are getting deployed on the same task manager

 

Hi everybody. We're currently experimenting the same characteristic on flink-1.6.2.

I've been reading that Flink treats all the slot as equals, it doesn't even know where these slots reside https://stackoverflow.com/questions/54980104/uneven-assignment-of-tasks-to-workers-in-flink.
So it should not be an issue; thus, the fact that it runs all the slots of a machine before moving to a new one should be just a rough coincidence.

Given that, I'm pretty sure that I've never been recording this feature using previous majors (I recall flink-1.3 for sure).
Moreover, this is damaging because you can get resources exhausted (e.g. memory, disk).

Hope we might find a solution on this.
Sincerely, 

Andrea

 

Il giorno lun 18 mar 2019 alle ore 11:53 Kumar Bolar, Harshith <[hidden email]> ha scritto:

Hi all,

 

We're running a Flink on a five node standalone cluster with three task manager (TM1, TM2, TM3) and two job managers.

 

Whenever I submit a new job, the job gets deployed on only TM3. When the number of slots in TM3 get exhausted, the jobs start getting deployed on TM2 and so on. How do I ensure that the jobs get distributed evenly across all 3 task managers?

 

Thanks,

Harshith

 

 

--

Andrea Spina
Software Engineer @ Radicalbit Srl
Via Borsieri 41, 20159, Milano - IT

[Andrey Zagrebin-3]

Hi Kumar and Andrea,

this is a known change in Flink behaviour from 1.4 to 1.5 (after FLIP-6).

There is an issue to track progress on more fine-grained task distribution [1].

Best,

Andrey

[1] https://issues.apache.org/jira/browse/FLINK-11815

On Mon, Mar 18, 2019 at 1:28 PM Kumar Bolar, Harshith <[hidden email]> wrote:

We recently upgraded to 1.7.2. Previously in 1.4.2 the jobs were getting deployed in a round robin manner. First job goes to TM1, second job goes to TM2 and so on. But after upgrading to 1.7.2, all jobs are going to the same task manager until the slots there run out. Did something change in 1.7?

 

Thanks,

Harshith

 

From: Andrea Spina <[hidden email]>
Date: Monday, 18 March 2019 at 5:54 PM
To: Harshith Kumar Bolar <[hidden email]>
Cc: user <[hidden email]>
Subject: [External] Re: Flink 1.7.2: All jobs are getting deployed on the same task manager

 

Hi everybody. We're currently experimenting the same characteristic on flink-1.6.2.

I've been reading that Flink treats all the slot as equals, it doesn't even know where these slots reside https://stackoverflow.com/questions/54980104/uneven-assignment-of-tasks-to-workers-in-flink.
So it should not be an issue; thus, the fact that it runs all the slots of a machine before moving to a new one should be just a rough coincidence.

Given that, I'm pretty sure that I've never been recording this feature using previous majors (I recall flink-1.3 for sure).
Moreover, this is damaging because you can get resources exhausted (e.g. memory, disk).

Hope we might find a solution on this.
Sincerely, 

Andrea

 

Il giorno lun 18 mar 2019 alle ore 11:53 Kumar Bolar, Harshith <[hidden email]> ha scritto:

Hi all,

 

We're running a Flink on a five node standalone cluster with three task manager (TM1, TM2, TM3) and two job managers.

 

Whenever I submit a new job, the job gets deployed on only TM3. When the number of slots in TM3 get exhausted, the jobs start getting deployed on TM2 and so on. How do I ensure that the jobs get distributed evenly across all 3 task managers?

 

Thanks,

Harshith

 

 

--

Andrea Spina
Software Engineer @ Radicalbit Srl
Via Borsieri 41, 20159, Milano - IT