Re: threading and distribution
Posted by
Matthias on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/threading-and-distribution-tp41296p41448.html
Hi Marco,
sorry for the late reply. The documentation you found [1] is already a good start. You can define how many subtasks of an operator run in parallel using the operator's parallelism configuration [2]. Each operator's subtask will run in a separate task slot. There's the concept of slot sharing as described in [3] which enables Flink to run subtasks of different operators of the same job in the same slot. This enables the TaskManager to run an entire pipeline in a single slot [3].
The maximum parallelism of your job is bound by the number of available task slots in the Flink cluster which can be defined through the number of slots per TaskManager [4][5] and the number of TaskManagers running in your Flink cluster (taskmanager.numberOfTaskSlots * #taskmanagers = maximum possible parallelism for an operator/pipeline).
I hope this was still helpful.
Best,
Matthias
On Fri, Feb 5, 2021 at 12:22 PM Marco Villalobos <
[hidden email]> wrote:
Okay, I am following up to my question. I see information regarding the threading and distribution model on the documentation about the architecture.
Next, I want to read up on what I have control over.
On Fri, Feb 5, 2021 at 3:06 AM Marco Villalobos <
[hidden email]> wrote:
as data flows from a source through a pipeline of operators and finally sinks, is there a means to control how many threads are used within an operator, and how an operator is distributed across the network?
Where can I read up on these types of details specifically?