How Flink decide which slots to use?

[Renkai]

I have 8 machine and I started a taskmanager with 24 slots on every machine.At first, I submited two apps, each of them used 8 slots, the all consumed one slot on each machine in the cluster.

Then I submited another application, but it behaves different, it used 8 slots,too. But 4 of them are run on the same machine.

Since this application used a lot of memory, such dispatch may cause a memory overflow. I want to know if I can custom the slot dispatcher.

[Aljoscha Krettek]

+Till since he is working a lot in these parts.

Could you maybe shed some light on this?

On Mon, 28 Nov 2016 at 11:34 Renkai Ge <[hidden email]> wrote:

I have 8 machine and I started a taskmanager with 24 slots on every machine.At first, I submited two apps, each of them used 8 slots, the all consumed one slot on each machine in the cluster.

Then I submited another application, but it behaves different, it used 8 slots,too. But 4 of them are run on the same machine.

Since this application used a lot of memory, such dispatch may cause a memory overflow. I want to know if I can custom the slot dispatcher.

[rmetzger0]

While we are waiting for Till's response, I can recommend this documentation page: https://ci.apache.org/projects/flink/flink-docs-release-1.2/concepts/index.html#workers-slots-resources 

On Thu, Dec 1, 2016 at 12:59 PM, Aljoscha Krettek <[hidden email]> wrote:

+Till since he is working a lot in these parts.

Could you maybe shed some light on this?

On Mon, 28 Nov 2016 at 11:34 Renkai Ge <[hidden email]> wrote:

I have 8 machine and I started a taskmanager with 24 slots on every machine.At first, I submited two apps, each of them used 8 slots, the all consumed one slot on each machine in the cluster.

Then I submited another application, but it behaves different, it used 8 slots,too. But 4 of them are run on the same machine.

Since this application used a lot of memory, such dispatch may cause a memory overflow. I want to know if I can custom the slot dispatcher.

[Stephan Ewen]

There is no deterministic rule what slots are picked by the scheduler.

For tasks that read data from other tasks, Flink will try to respect locality. Otherwise, the choice is "first that is free".

Best,

Stephan

On Mon, Dec 5, 2016 at 12:07 PM, Robert Metzger <[hidden email]> wrote:

While we are waiting for Till's response, I can recommend this documentation page: https://ci.apache.org/projects/flink/flink-docs-release-1.2/concepts/index.html#workers-slots-resources 

On Thu, Dec 1, 2016 at 12:59 PM, Aljoscha Krettek <[hidden email]> wrote:

+Till since he is working a lot in these parts.

Could you maybe shed some light on this?

On Mon, 28 Nov 2016 at 11:34 Renkai Ge <[hidden email]> wrote:

I have 8 machine and I started a taskmanager with 24 slots on every machine.At first, I submited two apps, each of them used 8 slots, the all consumed one slot on each machine in the cluster.

Then I submited another application, but it behaves different, it used 8 slots,too. But 4 of them are run on the same machine.

Since this application used a lot of memory, such dispatch may cause a memory overflow. I want to know if I can custom the slot dispatcher.