Filter with large key set

Posted by Eleanore Jin on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Filter-with-large-key-set-tp32167.html

Hi there, 

I have the following usecase:

a key set say [A,B,C,....] with around 10M entries, the type of the entries can be one of the types in BasicTypeInfo, e.g. String, Long, Integer etc...

and each message looks like below:

message: {

   header: A

   body: {}

}

I would like to use Flink to filter each message' header field, to see if the value present in the key set. 

The key set needs to be dynamic, meaning at any time, we can perform add/read/delete operations on the key set. 

Any suggestions are very welcome!

Thanks a lot!

Eleanore