How does Job manager schedule jobs to task managers

[Hung]

Hi Flink users,

I saw our Flink cluster doesn't dispatch jobs equally.
So some nodes are busy and have few task slots left, and some nodes are not busy and have many task slots.

We don't have yarn setup. Would that be the reason?

Or does Job manager dispatch job first to some nodes are not busy(more memory left and task slots)? or Job manager doesn't consider it?

Best,

Sendoh

[Till Rohrmann]

Hi Sendoh,

Flink does not necessarily spread the tasks across all available TMs. It depends a little bit on the topology of the job, because in general it is beneficial to deploy downstream tasks to the same machines where their input tasks run. I described the process in more detail here [1]. This link [2] also contains some information regarding Flink's internal scheduling.

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Scheduling-task-slots-in-round-robin-tp12068p12143.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.2/internals/job_scheduling.html

Cheers,

Till

On Tue, May 23, 2017 at 6:03 PM, Sendoh <[hidden email]> wrote:

Hi Flink users,

I saw our Flink cluster doesn't dispatch jobs equally.
So some nodes are busy and have few task slots left, and some nodes are not
busy and have many task slots.

We don't have yarn setup. Would that be the reason?

Or does Job manager dispatch job first to some nodes are not busy(more
memory left and task slots)? or Job manager doesn't consider it?

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-does-Job-manager-schedule-jobs-to-task-managers-tp13274.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.