Stream enrichment with static data, side inputs for DataStream

[Artur Mrozowski]

Hi, 

I have a stream of buildings and each building has foreign key reference to municipality. Municipalities data set is quite static. Both are placed on Kafka topics. I want to enrich each building with municipality name.   

FLIP 17, proposal would be ideal for this use case but it's still just a proposal 

https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API . 

My question is how do you work around issues like this? 

Best regards

Artur

[Averell]

Hi Artur,

Is that possible to make that "static" stream a keyedStream basing on that
foreign key?
If yes, then just connect the two streams, keyed on that foreign key. In the
CoProcessFunction, for every single record from the static stream, you write
its content into a ValueState; and for every record from the main stream,
you read the enrichment data from the saved ValueState to enrich that
mainstream record.

If no, then I am having the same issue :D Looking at Broadcast State, but
there is still something that doesn't look right for me.

Regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Artur Mrozowski]

Hi Averell,

thank you for your answer.  Indeed it is what I did. And it worked just fine in this case. I am sure I'll be facing mor complex scenarios, so it'd be very interesting to hear more about what you try to achieve by using broadcast state. 

Regards

On Thu, Feb 21, 2019 at 11:36 AM Averell <[hidden email]> wrote:

Hi Artur,

Is that possible to make that "static" stream a keyedStream basing on that
foreign key?
If yes, then just connect the two streams, keyed on that foreign key. In the
CoProcessFunction, for every single record from the static stream, you write
its content into a ValueState; and for every record from the main stream,
you read the enrichment data from the saved ValueState to enrich that
mainstream record.

If no, then I am having the same issue :D Looking at Broadcast State, but
there is still something that doesn't look right for me.

Regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/