Query regarding HA mode and checkpointing

[Kaushal Raj]

Hello,

I am new to flink. Have few queries regarding the HA mode with zookeeper and checkpointing. When flink is configured in HA mode with zookeeper, where do the job checkpoints are stored? zookeeper only used for recovering the jobmanager or even checkpoints? what is the significance of "high-availability.storageDir" here.

Thanks,

Kaushal

[Yang Wang]

Hi Kaushal,

Only the state handle pointer is stored in the ZooKeeper node. Since ZooKeeper is built for 

small data(KB level) storage. The real data will be persisted in the "high-availability.storageDir".

Note that it should be distributed storage(HDFS, S3, etc.).

The ZooKeeper HA service has the following usage, as well as the Kubernetes HA service,

you could find more information here[1].

* Leader election/retrieval

* Running job registry

* Submitted job graph store

* Checkpoint store

[1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-144%3A+Native+Kubernetes+HA+for+Flink#FLIP144:NativeKubernetesHAforFlink-HAcomponents

Best,

Yang

Kaushal Raj <[hidden email]> 于2020年12月3日周四 下午7:13写道：

Hello,

I am new to flink. Have few queries regarding the HA mode with zookeeper and checkpointing. When flink is configured in HA mode with zookeeper, where do the job checkpoints are stored? zookeeper only used for recovering the jobmanager or even checkpoints? what is the significance of "high-availability.storageDir" here.

Thanks,

Kaushal