Re: Forward Partitioning & same Parallelism: 1:1 communication?

Posted by Ufuk Celebi on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Forward-Partitioning-same-Parallelism-1-1-communication-tp2373p2377.html

Thanks :) Regarding your answer to Nica: I didn't mean to say that it was too generic or anything... it was very nice. I was just curious, that's why I asked.

On Wed, Aug 12, 2015 at 11:45 AM, Márton Balassi <[hidden email]> wrote:

Hey Ufuk,

The shipping strategy name forward is shared between batch and streaming and Nica did not specify either API, so I tried to give a generic answer.

I assume that your question is specifically for streaming, in that case: Yes, streaming is using the pointwise distribution pattern. [1] Unfortunately your concern is true, currently streaming would leave extra downstream operator instances idle, but Aljoscha has an open pull request fixing this issue amongst others. See the discussion here. [2]

[1] https://github.com/apache/flink/blob/master/flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java#L320

[2] https://github.com/apache/flink/pull/988

Cheers,

Marton

On Wed, Aug 12, 2015 at 11:33 AM, Ufuk Celebi <[hidden email]> wrote:

Hey Marton,

out of curiosity: is this using Flink’s “point” connections underneath or is there some custom logic for streaming jobs?

What happens if operator B has 2 times the parallelism of operator A? For example if there were parallel tasks A1 and A2 and B1-B4: would A1 send to B1 *and* B2 or just B1?

– Ufuk

On 12 Aug 2015, at 10:39, Márton Balassi <[hidden email]> wrote:

Dear Nica,

Yes, forward partitioning means that if subsequent operators share parallelism then the output of an upstream operator is sent to exactly one downstream operator. This makes sense for operators working on individual records, e.g. a typical map-filter pair, because as a consequence Flink may be able to collocate these operator pairs on the same physical machine.

Best,

Marton

On Tue, Aug 11, 2015 at 11:41 PM, Nicaz <[hidden email]> wrote:
Hello,

I have a question about forward partitioning in Flink.

If Operator A and Operator B have the same parallelism set and forward
partitioning is used for events coming from instances of A and going to
instances of B:

Will each instance of A send events to _exactly one_ instance of B?

That is, will all events coming from a specific instance of A go to the
_same_ specific instance of B, and will _all_ instances of B be used?
Or are there any situations where an instance of A will distribute events to
several different instances of B, or where two instances of A will send
events to the same instance of B (possibly leaving some other instance of B
unused)?

I'd be very happy if someone were able to shed some light on this issue. :-)

Thanks in advance
Nica



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Forward-Partitioning-same-Parallelism-1-1-communication-tp2373.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.