BroadcastStream vs Broadcasted DataStream

[Pieter-Jan Van Aeken]

Hi,

I am not sure I fully understand the differences between doing something like

dataStreamX.connect(dataStreamY.broadcast()).process(new CoProcessFunction{})

and this

dataStreamX.connect(dataStreamY.broadcast(descriptor).process(new BroadcastProcessFunction)

Couldn't I manage the state exactly the same way using the first option? The only difference I could find was in read/write permissions for different states. But other than that, the difference in possibilities escapes me and I was hoping someone here could clarify.

Kind regards,

Pieter-Jan

♻ Be green, keep it on the screen

[Kostas Kloudas]

Hi Pieter-Jan,

The second variant stores the elements of the broadcasted stream in operator (thus non-keyed) state.

On the differences: 

The Broadcast stream is not a keyed stream, so you are not in a keyed context, thus you have no access to keyed state.

Given this, and assuming that you are implementing a CoProcessFunction, then your function should also implement the 

ListCheckpointed interface and store the broadcasted data as ListState.

From a systems perspective, in case of rescaling, then your function should take care of removing duplicated manually

as in the case of scaling down for example, the state of 2 nodes may end up on one. In addition, given that elements are

re-distributed randomly, you may even end up with some elements missing one some nodes. This would be more evident 

in case of scaling up. In this case, nodes will be missing data.

From an API perspective, assuming that the other stream is keyed and it has state, accessing it in order to do some computation

would be impossible.

Cheers,

Kostas

On Oct 9, 2018, at 2:52 PM, Pieter-Jan Van Aeken <[hidden email]> wrote:

Hi,

I am not sure I fully understand the differences between doing something like

dataStreamX.connect(dataStreamY.broadcast()).process(new CoProcessFunction{})

and this

dataStreamX.connect(dataStreamY.broadcast(descriptor).process(new BroadcastProcessFunction)

Couldn't I manage the state exactly the same way using the first option? The only difference I could find was in read/write permissions for different states. But other than that, the difference in possibilities escapes me and I was hoping someone here could clarify.

Kind regards,

Pieter-Jan

♻ Be green, keep it on the screen