questions about YARN deployment and HDFS integration

[Chang Liu]

Dear All,

I am helping my team setup a Flink cluster and we would like to have high availability and easy to scale.

We would like to setup a minimal cluster environment but can be easily scaled in the future. This is my simple proposal: 

• 2 nodes
• each node is running a Flink instance, a YARN, and a HDFS
• Flink, YARN and HDFS are all running in cluster mode.

Based on it, my questions are:

1. By using HDFS as the file system, we can achieve fault tolerant (by recovering the checkpoint states when job fails). Question: so Flink itself is not capable of keeping and maintaining distributed state persistence just using local Linux file system, right?
2. Then, my follow-up is: if you have a Flink cluster (multiple nodes), and you use local Linux file system keeping the state checkpoints, what will happen if Flink job failed and Flink start to restart the job and recover the state from checkpoints?
3. If the Flink is deployed and managed on YARN, does that mean: if YARN is down, Flink is down?
4. If we have Flink managed by YARN, what is the purpose of using ZooKeeper? I did not really understand this part: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/jobmanager_high_availability.html. My question is: why YARN cannot provide this JobManager HA, and we have to add ZooKeeper?
5. How do you think I can keep different components of the architecture in different nodes (servers)? Do I keep every instance of Flink/YARN/HDFS on every single server, or I put each of them on completely different servers. Some of my considerations:
• if we put them on different servers, there will be many latency over the network between Flink <-> HDFS, and YARN <-> HDFS
• But if I each all of the 3 components Flink/YARN/HDFS on every server, they can also fight against each other for resources, right?
6. Correct me if i am wrong: one thing for sure is that, for every new where there is a Flink instance running, there should be a YARN running right?

Many thanks in advance!

Best regards/祝好,

Chang Liu 刘畅

[Kostas Kloudas]

Hi Chiang,

Some of the answers you can find in line:

On Sep 17, 2018, at 3:47 PM, Chang Liu <[hidden email]> wrote:

Dear All,

I am helping my team setup a Flink cluster and we would like to have high availability and easy to scale.

We would like to setup a minimal cluster environment but can be easily scaled in the future. This is my simple proposal: 

• 2 nodes
• each node is running a Flink instance, a YARN, and a HDFS
• Flink, YARN and HDFS are all running in cluster mode.

<image.jpeg>

Based on it, my questions are:

1. By using HDFS as the file system, we can achieve fault tolerant (by recovering the checkpoint states when job fails). Question: so Flink itself is not capable of keeping and maintaining distributed state persistence just using local Linux file system, right?

2. Then, my follow-up is: if you have a Flink cluster (multiple nodes), and you use local Linux file system keeping the state checkpoints, what will happen if Flink job failed and Flink start to restart the job and recover the state from checkpoints?

For both the above:

When a task fails, the whole job (all the tasks) are restarted, and are rescheduled on different machines.

If you use a local FS and you try to fetch state remotely upon recovery, how would the new nodes be able to locate

the state on a remote machine?

This is why Flink uses a distributed file system.

3. If the Flink is deployed and managed on YARN, does that mean: if YARN is down, Flink is down?

Well, it depends on which component fails. And I am not sure about all of them, but you could try it and see.

4. If we have Flink managed by YARN, what is the purpose of using ZooKeeper? I did not really understand this part: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/jobmanager_high_availability.html. My question is: why YARN cannot provide this JobManager HA, and we have to add ZooKeeper?

YARN can make sure that a new job master starts, but that master will have to fetch the state of the previous job master in order to know which jobs are running, their

progress, etc.

5. How do you think I can keep different components of the architecture in different nodes (servers)? Do I keep every instance of Flink/YARN/HDFS on every single server, or I put each of them on completely different servers. Some of my considerations:
• if we put them on different servers, there will be many latency over the network between Flink <-> HDFS, and YARN <-> HDFS
• But if I each all of the 3 components Flink/YARN/HDFS on every server, they can also fight against each other for resources, right?

You are right that you have to consider the above before deciding on your setup.

6. Correct me if i am wrong: one thing for sure is that, for every new where there is a Flink instance running, there should be a YARN running right?

Many thanks in advance!

Best regards/祝好,

Chang Liu 刘畅

I hope this helps,

Kostas

[Chang Liu]

Thanks for your answers :)

Best regards/祝好,

Chang Liu 刘畅

On 17 Sep 2018, at 17:25, Kostas Kloudas <[hidden email]> wrote:

Hi Chiang,

Some of the answers you can find in line:

On Sep 17, 2018, at 3:47 PM, Chang Liu <[hidden email]> wrote:

Dear All,

I am helping my team setup a Flink cluster and we would like to have high availability and easy to scale.

We would like to setup a minimal cluster environment but can be easily scaled in the future. This is my simple proposal: 

• 2 nodes
• each node is running a Flink instance, a YARN, and a HDFS
• Flink, YARN and HDFS are all running in cluster mode.

<image.jpeg>

Based on it, my questions are:

1. By using HDFS as the file system, we can achieve fault tolerant (by recovering the checkpoint states when job fails). Question: so Flink itself is not capable of keeping and maintaining distributed state persistence just using local Linux file system, right?

2. Then, my follow-up is: if you have a Flink cluster (multiple nodes), and you use local Linux file system keeping the state checkpoints, what will happen if Flink job failed and Flink start to restart the job and recover the state from checkpoints?

For both the above:

When a task fails, the whole job (all the tasks) are restarted, and are rescheduled on different machines.

If you use a local FS and you try to fetch state remotely upon recovery, how would the new nodes be able to locate

the state on a remote machine?

This is why Flink uses a distributed file system.

3. If the Flink is deployed and managed on YARN, does that mean: if YARN is down, Flink is down?

Well, it depends on which component fails. And I am not sure about all of them, but you could try it and see.

4. If we have Flink managed by YARN, what is the purpose of using ZooKeeper? I did not really understand this part: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/jobmanager_high_availability.html. My question is: why YARN cannot provide this JobManager HA, and we have to add ZooKeeper?

YARN can make sure that a new job master starts, but that master will have to fetch the state of the previous job master in order to know which jobs are running, their

progress, etc.

5. How do you think I can keep different components of the architecture in different nodes (servers)? Do I keep every instance of Flink/YARN/HDFS on every single server, or I put each of them on completely different servers. Some of my considerations:
• if we put them on different servers, there will be many latency over the network between Flink <-> HDFS, and YARN <-> HDFS
• But if I each all of the 3 components Flink/YARN/HDFS on every server, they can also fight against each other for resources, right?

You are right that you have to consider the above before deciding on your setup.

6. Correct me if i am wrong: one thing for sure is that, for every new where there is a Flink instance running, there should be a YARN running right?

Many thanks in advance!

Best regards/祝好,

Chang Liu 刘畅

I hope this helps,

Kostas