Parallelism vs task manager allocation

[pushpendra.jaiswal]

Hi

I have launched 2 task managers each with 2 slots. I have set parallelism 2 for one operator.

This operator should launch in both task managers with 1 slot each. 

But it is launching on only 1 task manager.

Is this intended behavior? 

Thanks

Pushpendra Jaiswal.

[Greg Hogan]

Hi Pushpendra,

This is the expected system behavior. Slots local to the same TaskManager can transfer buffers in memory. Are you able to also run the Sink with a parallelism of 4?

Greg

On Tue, Sep 20, 2016 at 6:16 AM, Pushpendra Jaiswal <[hidden email]> wrote:

Hi

I have launched 2 task managers each with 2 slots. I have set parallelism 2 for one operator.

This operator should launch in both task managers with 1 slot each. 

But it is launching on only 1 task manager.

Is this intended behavior? 

Thanks

Pushpendra Jaiswal.

[pushpendra.jaiswal]

Hi Greg
My sink is a hashmap(acting as db store), I am reading from this hashmap from other query stream. I need one instance / slot per task manager of sink, so that everyone refers the same instance/slot.
So is there any way so that I can restrict 1 slot per task manager.

Thanks
Pushpendra Jaiswal

[Greg Hogan]

Is the query stream also a Flink job? Is this use case not supported by keeping state within a single Flink job?
  https://ci.apache.org/projects/flink/flink-docs-master/dev/state.html

FLINK-3779 recently added "queryable state" to allow external processes access to operator state.
  https://issues.apache.org/jira/browse/FLINK-3779

On Wed, Sep 21, 2016 at 12:19 AM, pushpendra.jaiswal <[hidden email]> wrote:

Hi Greg
My sink is a hashmap(acting as db store), I am reading from this hashmap
from other query stream. I need one instance / slot per task manager of
sink, so that everyone refers the same instance/slot.
So is there any way so that I can restrict 1 slot per task manager.

Thanks
Pushpendra Jaiswal



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Parallelism-vs-task-manager-allocation-tp9084p9105.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.