questions about broadcasts

[Marco Villalobos-2]

Is it possible for an operator to receive two different kinds of broadcasts?  

Is it possible for an operator to receive two different types of streams and a broadcast? For example, I know there is a KeyedCoProcessFunction, but is there a version of that which can also receive broadcasts?

[Yun Gao]

Hi Marco,

(a) It is possible for an operator to receive two different kind of broadcasts, 

DataStream<Integer> ints = ....

DataStream<String> strs = ...

ints.broadcast().connect(strs.broadcast())

    ​.process(new CoProcessFunction<Integer, String, String>(){...});

(b) Traditional Flink operator could not accept three different inputs. 

There is a new MultipleInputOperator that could accept arbitrary number

of inputs [1]. However It is currently not expose directly to end users,

and you would need to work on some low-level api to use it. Or an alternative

might be use a tag to union the two input streams (or any two of the three inputs)

and use the (keyed)CoProcessFunction above. Also note that the broadcast is only a partitioner, 

and it is treated no difference with other partitioners for downstream operators.

Best,

Yun

[1] https://github.com/apache/flink/blob/51524de8fd337aafd30952873b36216c5a3c43bc/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorTest.java#L261

------------------------------------------------------------------
Sender:Marco Villalobos<[hidden email]>
Date:2021/03/06 09:47:53
Recipient:user<[hidden email]>
Theme:questions about broadcasts

Is it possible for an operator to receive two different kinds of broadcasts?  

Is it possible for an operator to receive two different types of streams and a broadcast? For example, I know there is a KeyedCoProcessFunction, but is there a version of that which can also receive broadcasts?