|
|
We have a task to handle stream order data. In one step ,we need fill to user name to stream data with lookup result in static configuration data by user id.
But, the user data is very big (100 million or more ).
Put all data in memory by key-value format will consume a great deal memory in every server in flink cluster and depress the performance of flink task.
We try to lookup data in memory-grid cache ,like redis & hazelcast. After test, performance result is not very well(about 250000/s per server). This result is below our design target(250000/s per server).
Is there any better solution to resolve this problem?
|
Locked
