Questions about slot sharing

[Vincent Wang]

Hi all,

  I'm totally new to Flink and I got a question about Flink's slot sharing feature.

  Suppose I have a pipeline like Mapper-> Reducer, and of cause the operators are not chained to a single task.

  When the job is actually scheduled to the Flink cluster, there is a Mapper task and a Reducer task scheduled to the same TaskManager's slot, here slot sharing works.

  The question is:  the slot sharing is an abstract concept, sharing slot doesn't means sharing thread, is that right? The Mapper task and Reducer task are still launched in independent threads.

Thanks in advance.

Thanks,

Huafeng   

  

[rmetzger0]

Hi Huafeng,

yes, the mapper and reducer are running in different threads on the TaskManager. Slot sharing is an abstract concept of the scheduler.

Flink also supports thread-sharing, for example when you have a series of mappers (or filters) running with the same parallelism. We call this feature "operator chaining" [1]

Regards,

Robert

[1] https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#tasks--operator-chains

On Thu, Jul 14, 2016 at 10:37 AM, Vincent Wang <[hidden email]> wrote:

Hi all,

  I'm totally new to Flink and I got a question about Flink's slot sharing feature.

  Suppose I have a pipeline like Mapper-> Reducer, and of cause the operators are not chained to a single task.

  When the job is actually scheduled to the Flink cluster, there is a Mapper task and a Reducer task scheduled to the same TaskManager's slot, here slot sharing works.

  The question is:  the slot sharing is an abstract concept, sharing slot doesn't means sharing thread, is that right? The Mapper task and Reducer task are still launched in independent threads.

Thanks in advance.

Thanks,

Huafeng   

  

[Vincent Wang]

Got it, thank you, Robert.

2016-07-14 20:55 GMT+08:00 Robert Metzger <[hidden email]>:

Hi Huafeng,

yes, the mapper and reducer are running in different threads on the TaskManager. Slot sharing is an abstract concept of the scheduler.

Flink also supports thread-sharing, for example when you have a series of mappers (or filters) running with the same parallelism. We call this feature "operator chaining" [1]

Regards,

Robert

[1] https://ci.apache.org/projects/flink/flink-docs-master/concepts/concepts.html#tasks--operator-chains

On Thu, Jul 14, 2016 at 10:37 AM, Vincent Wang <[hidden email]> wrote:

Hi all,

  I'm totally new to Flink and I got a question about Flink's slot sharing feature.

  Suppose I have a pipeline like Mapper-> Reducer, and of cause the operators are not chained to a single task.

  When the job is actually scheduled to the Flink cluster, there is a Mapper task and a Reducer task scheduled to the same TaskManager's slot, here slot sharing works.

  The question is:  the slot sharing is an abstract concept, sharing slot doesn't means sharing thread, is that right? The Mapper task and Reducer task are still launched in independent threads.

Thanks in advance.

Thanks,

Huafeng