HA and zookeeper

[Boris Lublinsky]

For HA implementation, is zookeeper is used only for leader selection, or it also stores some data relevant for switching to backup server

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Fabian Hueske-2]

Hi Boris,

ZooKeeper is also used by the JobManager to store metadata about the running job.

The JM writes information like the JobGraph, JAR file, checkpoint metadata to a persistent storage (like HDFS, S3, ...) and a pointer to this information to ZooKeeper.

In case of a recovery, the new JM looks up the pointer from ZooKeeper and fetches the job metadata from the persistent storage.

Best, Fabian

Am Sa., 6. Apr. 2019 um 01:28 Uhr schrieb Boris Lublinsky <[hidden email]>:

For HA implementation, is zookeeper is used only for leader selection, or it also stores some data relevant for switching to backup server

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Boris Lublinsky]

Thanks.

Is there:

1.  Documentation, describing this?

2. Any proposals/work trying to store it elsewhere?

The reason for this question is kubernetes deployment, where the use of zookeeper seems an overkill, but it will not work without zookeeper, see https://jobs.zalando.com/tech/blog/running-apache-flink-on-kubernetes/?gh_src=4n3gxh1

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Apr 8, 2019, at 10:29 AM, Fabian Hueske <[hidden email]> wrote:

Hi Boris,

ZooKeeper is also used by the JobManager to store metadata about the running job.

The JM writes information like the JobGraph, JAR file, checkpoint metadata to a persistent storage (like HDFS, S3, ...) and a pointer to this information to ZooKeeper.

In case of a recovery, the new JM looks up the pointer from ZooKeeper and fetches the job metadata from the persistent storage.

Best, Fabian

Am Sa., 6. Apr. 2019 um 01:28 Uhr schrieb Boris Lublinsky <[hidden email]>:

For HA implementation, is zookeeper is used only for leader selection, or it also stores some data relevant for switching to backup server

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Konstantin Knauf-2]

Hi Boris,

I am not aware of documentation describing this in detail. There is an open JIRA for a High Availability Service based on etcd [1].

Cheers,

Konstantin

[1] https://issues.apache.org/jira/browse/FLINK-11105

On Mon, Apr 8, 2019 at 3:20 PM Boris Lublinsky <[hidden email]> wrote:

Thanks.

Is there:

1.  Documentation, describing this?

2. Any proposals/work trying to store it elsewhere?

The reason for this question is kubernetes deployment, where the use of zookeeper seems an overkill, but it will not work without zookeeper, see https://jobs.zalando.com/tech/blog/running-apache-flink-on-kubernetes/?gh_src=4n3gxh1

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Apr 8, 2019, at 10:29 AM, Fabian Hueske <[hidden email]> wrote:

Hi Boris,

ZooKeeper is also used by the JobManager to store metadata about the running job.

The JM writes information like the JobGraph, JAR file, checkpoint metadata to a persistent storage (like HDFS, S3, ...) and a pointer to this information to ZooKeeper.

In case of a recovery, the new JM looks up the pointer from ZooKeeper and fetches the job metadata from the persistent storage.

Best, Fabian

Am Sa., 6. Apr. 2019 um 01:28 Uhr schrieb Boris Lublinsky <[hidden email]>:

For HA implementation, is zookeeper is used only for leader selection, or it also stores some data relevant for switching to backup server

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

--