Checkpointing and save pointing

[Boris Lublinsky]

I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Congxian Qiu]

Hi, Boris

TM will also need to write to the external volume.

Best, Congxian

On May 8, 2019, 03:56 +0800, Boris Lublinsky <[hidden email]>, wrote:

I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Fabian Hueske-2]

Hi Boris,

Is your question is in the context of replacing Zookeeper by a different service for highly-available setups or are you setting up a regular Flink cluster?

Best, Fabian

Am Mi., 8. Mai 2019 um 06:20 Uhr schrieb Congxian Qiu <[hidden email]>:

Hi, Boris

TM will also need to write to the external volume.

Best, Congxian

On May 8, 2019, 03:56 +0800, Boris Lublinsky <[hidden email]>, wrote:

I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Boris Lublinsky]

For now is a regular Link cluster,

But even there I want to use both check and save pointing.

We do not want to use Hadoop, but rather shared fs - NFS/Gluster.

I was trying to see whether volumes need to be mounted only for Job manager or both.

HA is the next step. Trying to find the code saving checkpointing location.

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On May 10, 2019, at 2:47 AM, Fabian Hueske <[hidden email]> wrote:

Hi Boris,

Is your question is in the context of replacing Zookeeper by a different service for highly-available setups or are you setting up a regular Flink cluster?

Best, Fabian

Am Mi., 8. Mai 2019 um 06:20 Uhr schrieb Congxian Qiu <[hidden email]>:

Hi, Boris

TM will also need to write to the external volume.

Best, Congxian

On May 8, 2019, 03:56 +0800, Boris Lublinsky <[hidden email]>, wrote:

I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

[Fabian Hueske-2]

For checkpoints and savepoints, the JM and all TMs need access to the same storage system.

This can be shared NFS that is mounted on each machine.

Best, Fabian

Am Fr., 10. Mai 2019 um 15:15 Uhr schrieb Boris Lublinsky <[hidden email]>:

For now is a regular Link cluster,

But even there I want to use both check and save pointing.

We do not want to use Hadoop, but rather shared fs - NFS/Gluster.

I was trying to see whether volumes need to be mounted only for Job manager or both.

HA is the next step. Trying to find the code saving checkpointing location.

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On May 10, 2019, at 2:47 AM, Fabian Hueske <[hidden email]> wrote:

Hi Boris,

Is your question is in the context of replacing Zookeeper by a different service for highly-available setups or are you setting up a regular Flink cluster?

Best, Fabian

Am Mi., 8. Mai 2019 um 06:20 Uhr schrieb Congxian Qiu <[hidden email]>:

Hi, Boris

TM will also need to write to the external volume.

Best, Congxian

On May 8, 2019, 03:56 +0800, Boris Lublinsky <[hidden email]>, wrote:

I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?

Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/