Re: Task Slots and Heterogeneous Tasks

Posted by Stephan Ewen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Task-Slots-and-Heterogeneous-Tasks-tp6118p6125.html

Hi!

Slots are usually shared between the heavy and non heavy tasks, for that reason.

Have a look at these resources: https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#workers-slots-resources

Let us know if you have more questions!

Greetings,

Stephan

On Fri, Apr 15, 2016 at 1:20 AM, Maxim <[hidden email]> wrote:

I'm trying to understand a behavior of Flink in case of heterogeneous operations. For example in our pipelines some operation might accumulate large windows while another performs high latency calls to external services. Obviously the former needs task slot with a large memory allocation, while the latter needs no memory but a high degree of parallelism. 

Is any way to have different slot types and control allocation of operations to them? May be is there another way to ensure good hardware utilization?

Also from the documentation it is not clear if memory of a TaskManager is shared across all tasks running on it or each task gets its quota. Could you clarify it?

Thanks,

Maxim.