Re: Handling Large Broadcast States
Posted by
Piotr Nowojski-5 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Handling-Large-Broadcast-States-tp44493p44570.html
Hi,
As far as I know there are no plans to support other state backends with BroadcastState. I don't know about any particular technical limitation, it probably just hasn't been done. Also I don't know how much effort that would be. Probably it wouldn't be easy.
Timo, can you chip in how for example Table API/SQL is solving this problem? I'm pretty sure Tablie API is using broadcast joins after all?
Best,
Piotrek
Hey Flink folks,
I was discussing the use of the Broadcast Pattern with some colleagues today for a potential enrichment use-case and noticed that it wasn’t currently backed by RocksDB. This seems to indicate that it would be solely limited to the memory allocated, which might not support a large enrichment data set that our use case might run into (thousands of tenants with users and various other entities to enrich by).
Are there any plans to eventually add support for BroadcastState to be backed by a non-memory source? Or perhaps some technical limitations that might not make that possible? If the latter is true, is there a preferred pattern for handling enrichment/lookups for a very large set of data that may not be memory-bound?
Any advice or thoughts would be welcome!
Rion