Hi,
I'm working with two separate Kubernetes clusters located in different regions (hosted in proprietary data centers), the distance between the regions introduces a pretty high (~50ms) latency between the clusters, so communication should not go cross-site
unless necessary. I would like to use one Flink cluster spanning both K8s clusters. I use Rocksdb as state backend.
What would be an appropriate Flink setup for a K8s setup such as this? The current stream source is RabbitMQ (not hosted within K8s).
My first naive thought is to use a active-passive setup with one separate job manager in each cluster and separate task managers for each job manager. I.e. flink would only be active processing data in one cluster at a time. I'm guessing that ZooKeeper
would be necessary to enable leader election between the two clusters?
Any ideas or real world experience with similar setups? Does this sound like a viable thing to do or is it just not meant to be done this way?
Thanks,
Thomas
Locked
