How to Broadcast a very large model object (used in iterative scoring in recommendation system) in Flink

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

How to Broadcast a very large model object (used in iterative scoring in recommendation system) in Flink

Anchit Jatana
Hi All,

I'm building a recommendation system streaming application for which I need to broadcast a very large model object (used in iterative scoring) among all the task managers performing the operation parallely for the operator

I'm doing an this operation in map1 of CoMapFunction. Please suggest me some way to achieve the broadcasting of the large model variable (something similar to what Spark has with broadcast variables).

Thank you

Regards,
Anchit