Re: Task Slots and Heterogeneous Tasks

Posted by Till Rohrmann on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Task-Slots-and-Heterogeneous-Tasks-tp6118p6130.html

Hi Maxim,

concerning your second part of the question: The managed memory of a TaskManager is first split among the available slots. Each slot portion of the managed memory is again split among all operators which require managed memory when a pipeline is executed. In contrast to that, the heap memory is shared by all concurrently running tasks.

Cheers,

Till

On Fri, Apr 15, 2016 at 1:58 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

Slots are usually shared between the heavy and non heavy tasks, for that reason.

Have a look at these resources: https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#workers-slots-resources

Let us know if you have more questions!

Greetings,

Stephan

On Fri, Apr 15, 2016 at 1:20 AM, Maxim <[hidden email]> wrote:

I'm trying to understand a behavior of Flink in case of heterogeneous operations. For example in our pipelines some operation might accumulate large windows while another performs high latency calls to external services. Obviously the former needs task slot with a large memory allocation, while the latter needs no memory but a high degree of parallelism. 

Is any way to have different slot types and control allocation of operations to them? May be is there another way to ensure good hardware utilization?

Also from the documentation it is not clear if memory of a TaskManager is shared across all tasks running on it or each task gets its quota. Could you clarify it?

Thanks,

Maxim.