realtion between operator and task

[Radu Tudoran]

Hi,

 

Can someone explain how and where a stream operator is mapped to a stream task.

I am particularly interested in the way the stream outputs are created and attached to the operators. I saw that this happen in OperatorChain functions but I do not have the picture of the lifecycle of an stream operator that you would create to its mapping to the task and assignment of the output binding.

[Stephan Ewen]

Hi!

A "task" is something that is deployed as one unit to the TaskManager and runs in one thread.

A task can have multiple "operators" chained together, usually one per user function, for example "Map -> Filter -> FlatMap -> AssignTimestamps -> ..."

Stephan

On Mon, Mar 7, 2016 at 7:36 PM, Radu Tudoran <[hidden email]> wrote:

Hi,

 

Can someone explain how and where a stream operator is mapped to a stream task.

I am particularly interested in the way the stream outputs are created and attached to the operators. I saw that this happen in OperatorChain functions but I do not have the picture of the lifecycle of an stream operator that you would create to its mapping to the task and assignment of the output binding.

[Radu Tudoran]

Hi,

 

Thanks for the answer. Can you point me to the code where the operators are being assign to tasks.

Thanks

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Stephan Ewen
Sent: Monday, March 07, 2016 8:29 PM
To: [hidden email]
Subject: Re: realtion between operator and task

 

Hi!

 

A "task" is something that is deployed as one unit to the TaskManager and runs in one thread.

 

A task can have multiple "operators" chained together, usually one per user function, for example "Map -> Filter -> FlatMap -> AssignTimestamps -> ..."

 

Stephan

 

 

On Mon, Mar 7, 2016 at 7:36 PM, Radu Tudoran <[hidden email]> wrote:

Hi,

 

Can someone explain how and where a stream operator is mapped to a stream task.

I am particularly interested in the way the stream outputs are created and attached to the operators. I saw that this happen in OperatorChain functions but I do not have the picture of the lifecycle of an stream operator that you would create to its mapping to the task and assignment of the output binding.

 

[Till Rohrmann]

Hi Radu,

the mapping which StreamOperator is executed by which StreamTask happens first in the StreamGraph.addOperator method. However, there is a second step in the StreamingJobGraphGenerator.createChain where chainable operators are chained and then executed by a single StreamTask. The construction of the actual operator chain happens in the class OperatorChain.

Cheers,
Till

​

On Tue, Mar 8, 2016 at 10:51 AM, Radu Tudoran <[hidden email]> wrote:

Hi,

 

Thanks for the answer. Can you point me to the code where the operators are being assign to tasks.

Thanks

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Stephan Ewen
Sent: Monday, March 07, 2016 8:29 PM
To: [hidden email]
Subject: Re: realtion between operator and task

 

Hi!

 

A "task" is something that is deployed as one unit to the TaskManager and runs in one thread.

 

A task can have multiple "operators" chained together, usually one per user function, for example "Map -> Filter -> FlatMap -> AssignTimestamps -> ..."

 

Stephan

 

 

On Mon, Mar 7, 2016 at 7:36 PM, Radu Tudoran <[hidden email]> wrote:

Hi,

 

Can someone explain how and where a stream operator is mapped to a stream task.

I am particularly interested in the way the stream outputs are created and attached to the operators. I saw that this happen in OperatorChain functions but I do not have the picture of the lifecycle of an stream operator that you would create to its mapping to the task and assignment of the output binding.