Localenvironment jobcluster ha High availability

[Eric HOFFMANN-2]

Hi, i use a jobcluster (1 manager and 1 worker) in kubernetes for streaming application, i would like to have the lightest possible solution, is it possible to use a localenvironment (manager and worker embeded) and still have HA with zookeeper in this mode?, I mean kubernetes will restart the job, in the case of jobcluster, metadata are retrieve from zookeeper and data from S3 or hdfs, is this pattern the same in localenvironment ?
Thx
Eric

This message contains confidential information and is intended only for the individual(s) addressed in the message. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you are not the intended recipient, you are notified that disclosing, distributing, or copying this e-mail is strictly prohibited.

[Gary Yao-4]

Hi Eric,

What you say should be possible because your job will be executed in a
MiniCluster [1] which has HA support. I have not tried this out myself,
and I am not aware that people are doing this in production. However,
there are integration tests that use MiniCluster + ZooKeeper [2].

Best,
Gary

[1] https://github.com/apache/flink/blob/eddad99123525211c900102206384dacaf8385fc/flink-runtime/src/main/java/org/apache/flink/runtime/minicluster/MiniCluster.java
[2] https://github.com/apache/flink/blob/eddad99123525211c900102206384dacaf8385fc/flink-tests/src/test/java/org/apache/flink/test/checkpointing/EventTimeWindowCheckpointingITCase.java

On Tue, Dec 3, 2019 at 7:21 PM Eric HOFFMANN <[hidden email]> wrote:

Hi, i use a jobcluster (1 manager and 1 worker) in kubernetes for streaming application, i would like to have the lightest possible solution, is it possible to use a localenvironment (manager and worker embeded) and still have HA with zookeeper in this mode?, I mean kubernetes will restart the job, in the case of jobcluster, metadata are retrieve from zookeeper and data from S3 or hdfs, is this pattern the same in localenvironment ?
Thx
Eric

This message contains confidential information and is intended only for the individual(s) addressed in the message. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you are not the intended recipient, you are notified that disclosing, distributing, or copying this e-mail is strictly prohibited.