Flink and Nomad ( from Hashicorp)

[Vishal Santoshi]

Is there any reason not to have Nomad HA  on the lines of K8s HA ? I think it would depend on how puggable the HA core code is ? Any links to how ZK/K8s code specifically for HA would be highly appreciated....

[Till Rohrmann]

Hi Vishal,

There is no specific reason why Flink does not have a Nomad HA implementation other than it has not been done yet. As long as Nomad supports leader election, service recovery and persisting small data it should be possible to implement HighAvailabilityServices against it. The best pointers to get started would be the implementation of the HighAvailabilityServices [1,2].

If you would like to contribute such a feature, then please open a JIRA ticket for it.

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java

[2] https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesHaServices.java

Cheers,

Till

On Tue, Mar 9, 2021 at 7:56 PM Vishal Santoshi <[hidden email]> wrote:

Is there any reason not to have Nomad HA  on the lines of K8s HA ? I think it would depend on how puggable the HA core code is ? Any links to how ZK/K8s code specifically for HA would be highly appreciated....

[Vishal Santoshi]

Makes total sense, Thanks, I'll check it out. 

On Wed, Mar 10, 2021 at 1:28 PM Till Rohrmann <[hidden email]> wrote:

Hi Vishal,

There is no specific reason why Flink does not have a Nomad HA implementation other than it has not been done yet. As long as Nomad supports leader election, service recovery and persisting small data it should be possible to implement HighAvailabilityServices against it. The best pointers to get started would be the implementation of the HighAvailabilityServices [1,2].

If you would like to contribute such a feature, then please open a JIRA ticket for it.

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/zookeeper/ZooKeeperHaServices.java

[2] https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesHaServices.java

Cheers,

Till

On Tue, Mar 9, 2021 at 7:56 PM Vishal Santoshi <[hidden email]> wrote:

Is there any reason not to have Nomad HA  on the lines of K8s HA ? I think it would depend on how puggable the HA core code is ? Any links to how ZK/K8s code specifically for HA would be highly appreciated....