What happens when: high-availability.storageDir: is not available?

[John Smith]

In HA setup what would happen if high-availability.storageDir: was unavailable?

[John Smith]

Or even how often do the master write there?

On Fri, 7 Jun 2019 at 16:16, John Smith <[hidden email]> wrote:

In HA setup what would happen if high-availability.storageDir: was unavailable?

[Yun Tang]

High availability storage directory would store completed checkpoint and submitted job graph and completed checkpoint. If this directory is unavailable when initialization, job would be submitted well. If this directory is unavailable when creating checkpoints, that checkpoint would finally failed. The frequency of  this directory updated mainly depends on the checkpoint interval.

Best

Yun Tang

---

From: John Smith <[hidden email]>
Sent: Monday, June 10, 2019 23:55
To: user
Subject: Re: What happens when: high-availability.storageDir: is not available?

 

Or even how often do the master write there?

On Fri, 7 Jun 2019 at 16:16, John Smith <[hidden email]> wrote:

In HA setup what would happen if high-availability.storageDir: was unavailable?

[John Smith]

Thanks. Does this folder need to available for task nodes as well? Or just job nodes?

On Wed., Jun. 12, 2019, 11:56 p.m. Yun Tang, <[hidden email]> wrote:

High availability storage directory would store completed checkpoint and submitted job graph and completed checkpoint. If this directory is unavailable when initialization, job would be submitted well. If this directory is unavailable when creating checkpoints, that checkpoint would finally failed. The frequency of  this directory updated mainly depends on the checkpoint interval.

Best

Yun Tang

---

From: John Smith <[hidden email]>
Sent: Monday, June 10, 2019 23:55
To: user
Subject: Re: What happens when: high-availability.storageDir: is not available?

 

Or even how often do the master write there?

On Fri, 7 Jun 2019 at 16:16, John Smith <[hidden email]> wrote:

In HA setup what would happen if high-availability.storageDir: was unavailable?

[Yun Tang]

Except job graph and completed checkpoint, high availability storage directory would also store blob data which would be accessed from both jobmanager and taskmanager nodes, you could refer to [1] to view the BLOB storage architecture.

[1] <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A&#43;Improved&#43;BLOB&#43;storage&#43;architecture" id="LPNoLP207037"> https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture

Best

Yun Tang

---

From: John Smith <[hidden email]>
Sent: Thursday, June 13, 2019 21:18
To: Yun Tang
Cc: user
Subject: Re: What happens when: high-availability.storageDir: is not available?

 

Thanks. Does this folder need to available for task nodes as well? Or just job nodes?

On Wed., Jun. 12, 2019, 11:56 p.m. Yun Tang, <[hidden email]> wrote:

High availability storage directory would store completed checkpoint and submitted job graph and completed checkpoint. If this directory is unavailable when initialization, job would be submitted well. If this directory is unavailable when creating checkpoints, that checkpoint would finally failed. The frequency of  this directory updated mainly depends on the checkpoint interval.

Best

Yun Tang

---

From: John Smith <[hidden email]>
Sent: Monday, June 10, 2019 23:55
To: user
Subject: Re: What happens when: high-availability.storageDir: is not available?

 

Or even how often do the master write there?

On Fri, 7 Jun 2019 at 16:16, John Smith <[hidden email]> wrote:

In HA setup what would happen if high-availability.storageDir: was unavailable?