Apache Beam Side input vs Flink Broadcast Stream

[Eleanore Jin]

Hi All, 

there is a recent published article in the flink official website for running beam on top of flink https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html

In the article: 

• You get additional features like side inputs and cross-language pipelines that are not supported natively in Flink but only supported when using Beam with Flink

Ultimately, Beam pipeline will be translated into Flink job. So does beam's side input translates into Flink Broadcast stream? 

If I look at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators, it looks like converting the the side input into broadcast stream, then why it says Flink does not support it natively? 

Thanks a lot!

Eleanore

[Arvid Heise-3]

Hi Eleanore,

we understand side-input as something more general than simple broadcast input, see FLIP-17 for details [1].

If a broadcast fits your use case, you can use that of course. We are aiming for something, where a side input can also be co-partitioned. We are currently laying the foundations for that feature.

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API

On Thu, Feb 27, 2020 at 6:46 AM Jin Yi <[hidden email]> wrote:

Hi All, 

there is a recent published article in the flink official website for running beam on top of flink https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html

In the article: 

• You get additional features like side inputs and cross-language pipelines that are not supported natively in Flink but only supported when using Beam with Flink

Ultimately, Beam pipeline will be translated into Flink job. So does beam's side input translates into Flink Broadcast stream? 

If I look at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators, it looks like converting the the side input into broadcast stream, then why it says Flink does not support it natively? 

Thanks a lot!

Eleanore

[Eleanore Jin]

Hi Arvid, 

Thanks a lot for the response and yes I am aware of FLIP-17. 

Eleanore 

On Fri, Feb 28, 2020 at 2:16 AM Arvid Heise <[hidden email]> wrote:

Hi Eleanore,

we understand side-input as something more general than simple broadcast input, see FLIP-17 for details [1].

If a broadcast fits your use case, you can use that of course. We are aiming for something, where a side input can also be co-partitioned. We are currently laying the foundations for that feature.

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API

On Thu, Feb 27, 2020 at 6:46 AM Jin Yi <[hidden email]> wrote:

Hi All, 

there is a recent published article in the flink official website for running beam on top of flink https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html

In the article: 

• You get additional features like side inputs and cross-language pipelines that are not supported natively in Flink but only supported when using Beam with Flink

Ultimately, Beam pipeline will be translated into Flink job. So does beam's side input translates into Flink Broadcast stream? 

If I look at org.apache.beam.runners.flink.FlinkStreamingTransformTranslators, it looks like converting the the side input into broadcast stream, then why it says Flink does not support it natively? 

Thanks a lot!

Eleanore