Re: Using redis cache in flink

Posted by Yun Tang on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Using-redis-cache-in-flink-tp31999p32001.html

Hi Navneeth

If you wrap redis as a state backend, you cannot easily share data across slots as Flink construct state backend per operator with local thread only.

If you use a redis cluster as a externalized service to store your data, you can share data across slots easily. However, compared with the reduced cost of serialization, the introduce of network communicate cannot be ignored. There exists trade-off here, and we cannot ensure there would be a performance gain. Actually, I prefer the time used in CPU serialization is much less than the time consumed through the network.

Best

Yun Tang

---

From: Navneeth Krishnan <[hidden email]>
Sent: Wednesday, January 8, 2020 12:33
To: user <[hidden email]>
Subject: Using redis cache in flink

 

Hi All,

I want to use redis as near far cache to store data which are common across slots i.e. share data across slots. This data is required for processing every single message and it's better to store in a in memory cache backed by redis rather than rocksdb since it has to be serialized for every single get call. Do you guys think this is good solution or is there any other better solution? Also, Is there any reference on how I can create a centralized near far cache since the job and operators are distributed by the job manager.

Thanks