Fwd: Updating Broadcast Variables

[Pete Wiseman]

Hi Guys, 

Quick question about broadcast variables. 

How would one periodically update the value of a broadcast variable? 

Suppose my broadcast variable is used for lookups, and these lookup values can change over time (i.e. some of the lookup values get removed, whilst others get added). Is it possible to "rebroadcast" it's value? 

Or is there some other way to do this kind of thing? 

Many thanks, 

Pete

 

[Fabian Hueske-2]

Hi Pete,

Broadcast variables are a feature of the DataSet API [1], i.e., available for batch processing.

Broadcast variables are computed based on the complete input (which is possible because they are only available for bounded data sets and not for unbounded streams) and shared with all operator that require them.

Therefore, there is not need to update a broadcast variable because there is no more input to update the variable.

I assume that you are looking for a similar feature for the DataStream API.

In the DataStream API, you would use a function with two input, for example a CoProcessFunction, and broadcast one of the inputs.

Note that with the current version of Flink (1.4.2) you can only connect a non-keyed stream with a broadcasted stream. The upcoming Flink 1.5.0 will remove this limitation by introducing broadcasted state.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/batch/index.html#broadcast-variables

2018-04-04 5:38 GMT+02:00 Pete Wiseman <[hidden email]>:

Hi Guys, 

Quick question about broadcast variables. 

How would one periodically update the value of a broadcast variable? 

Suppose my broadcast variable is used for lookups, and these lookup values can change over time (i.e. some of the lookup values get removed, whilst others get added). Is it possible to "rebroadcast" it's value? 

Or is there some other way to do this kind of thing? 

Many thanks, 

Pete