Difference between JobManager and JobMaster

[Lu Weizheng]

Hi all,

Recently I am reading source code of Flink. There are both JobManager and JobMaster in flink-runtime project. And JobManagerRunner interface says it is a runner which executes a JobMaster. So how to distinguish the two concepts? Is JobMaster a subset of JobManager? Or JobMaster is a new version which is going to replace the function of JobManager? Or the two roles are different?

Thank you!

Weizheng

[Yun Tang]

Hi Wizheng

FLIP-6 [1] introduce a new implementation of master component in Flink cluster with name of `JobMaster.java`[2], you can find previous implementation in Flink-1.7 as `JobManager.scala` [3]. However, we would still call the master component as 'JobManager' though its implementation name has been changed.

If you want to know more details, please read FLIP-6 and its JIRA issue FLINK-4319 [4].

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077

[2] https://github.com/apache/flink/blob/release-1.10/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java

[3] https://github.com/apache/flink/blob/release-1.7/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala

[4] https://issues.apache.org/jira/browse/FLINK-4319

Best

Yun Tang

---

From: Lu Weizheng <[hidden email]>
Sent: Friday, January 31, 2020 10:25
To: [hidden email] <[hidden email]>
Subject: Difference between JobManager and JobMaster

 

Hi all,

Recently I am reading source code of Flink. There are both JobManager and JobMaster in flink-runtime project. And JobManagerRunner interface says it is a runner which executes a JobMaster. So how to distinguish the two concepts? Is JobMaster a subset of JobManager? Or JobMaster is a new version which is going to replace the function of JobManager? Or the two roles are different?

Thank you!

Weizheng

[Lu Weizheng]

Hi Yun, thank you so much for your reply!

I have briefly read all links in your reply. So the JobManager.scala was removed and JobMaster was added and functions what JobManager did before? 

Best Regards

Weizheng

---

发件人: Yun Tang <[hidden email]>
发送时间: 2020年1月31日 15:20
收件人: Lu Weizheng <[hidden email]>; [hidden email] <[hidden email]>
主题: Re: Difference between JobManager and JobMaster

 

Hi Wizheng

FLIP-6 [1] introduce a new implementation of master component in Flink cluster with name of `JobMaster.java`[2], you can find previous implementation in Flink-1.7 as `JobManager.scala` [3]. However, we would still call the master component as 'JobManager' though its implementation name has been changed.

If you want to know more details, please read FLIP-6 and its JIRA issue FLINK-4319 [4].

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077

[2] https://github.com/apache/flink/blob/release-1.10/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java

[3] https://github.com/apache/flink/blob/release-1.7/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala

[4] https://issues.apache.org/jira/browse/FLINK-4319

Best

Yun Tang

---

From: Lu Weizheng <[hidden email]>
Sent: Friday, January 31, 2020 10:25
To: [hidden email] <[hidden email]>
Subject: Difference between JobManager and JobMaster

 

Hi all,

Recently I am reading source code of Flink. There are both JobManager and JobMaster in flink-runtime project. And JobManagerRunner interface says it is a runner which executes a JobMaster. So how to distinguish the two concepts? Is JobMaster a subset of JobManager? Or JobMaster is a new version which is going to replace the function of JobManager? Or the two roles are different?

Thank you!

Weizheng

[Chesnay Schepler]

They are similar but not quite the same. The JobManager was responsible for managing multiple jobs, whereas the JobMaster only manages a single job. The responsibility for managing multiple jobs (==JobMasters) and accepting job submissions was moved into the Dispatcher.

On 31/01/2020 09:39, Lu Weizheng wrote:

Hi Yun, thank you so much for your reply!

I have briefly read all links in your reply. So the JobManager.scala was removed and JobMaster was added and functions what JobManager did before? 

Best Regards

Weizheng

---

发件人: Yun Tang [hidden email]
发送时间: 2020年1月31日 15:20
收件人: Lu Weizheng [hidden email]; [hidden email] [hidden email]
主题: Re: Difference between JobManager and JobMaster

 

Hi Wizheng

FLIP-6 [1] introduce a new implementation of master component in Flink cluster with name of `JobMaster.java`[2], you can find previous implementation in Flink-1.7 as `JobManager.scala` [3]. However, we would still call the master component as 'JobManager' though its implementation name has been changed.

If you want to know more details, please read FLIP-6 and its JIRA issue FLINK-4319 [4].

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077

[2] https://github.com/apache/flink/blob/release-1.10/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java

[3] https://github.com/apache/flink/blob/release-1.7/flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala

[4] https://issues.apache.org/jira/browse/FLINK-4319

Best

Yun Tang

---

From: Lu Weizheng [hidden email]
Sent: Friday, January 31, 2020 10:25
To: [hidden email] [hidden email]
Subject: Difference between JobManager and JobMaster

 

Hi all,

Recently I am reading source code of Flink. There are both JobManager and JobMaster in flink-runtime project. And JobManagerRunner interface says it is a runner which executes a JobMaster. So how to distinguish the two concepts? Is JobMaster a subset of JobManager? Or JobMaster is a new version which is going to replace the function of JobManager? Or the two roles are different?

Thank you!

Weizheng