Why does jobmanager running needs slot ?

[mingleizhang-2]

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.

 

 

[Piotr Nowojski]

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <[hidden email]> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.

 

 

[mingleizhang-2]

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?

Rice.

At 2018-02-05 17:30:32, "Piotr Nowojski" <[hidden email]> wrote:

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <[hidden email]> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.

 

 

 

[Piotr Nowojski]

I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).

Piotrek

On 5 Feb 2018, at 10:44, mingleizhang <[hidden email]> wrote:

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?

Rice.

At 2018-02-05 17:30:32, "Piotr Nowojski" <[hidden email]> wrote:

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <[hidden email]> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.

 

 

 

[mingleizhang-2]

 Makes sense to me now. Is it a new design at FLIP6 ?

Rice.

At 2018-02-05 17:49:05, "Piotr Nowojski" <[hidden email]> wrote:

I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).

Piotrek

On 5 Feb 2018, at 10:44, mingleizhang <[hidden email]> wrote:

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?

Rice.

At 2018-02-05 17:30:32, "Piotr Nowojski" <[hidden email]> wrote:

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <[hidden email]> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.

 

 

 

 

[Piotr Nowojski]

It seems so - but I’m saying this only basing on a annotations when this method was added (in the last couple of months). I’m not that much familiar with those code parts.

Piotrek

On 5 Feb 2018, at 10:51, mingleizhang <[hidden email]> wrote:

 Makes sense to me now. Is it a new design at FLIP6 ?

Rice.

At 2018-02-05 17:49:05, "Piotr Nowojski" <[hidden email]> wrote:

I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).

Piotrek

On 5 Feb 2018, at 10:44, mingleizhang <[hidden email]> wrote:

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?

Rice.

At 2018-02-05 17:30:32, "Piotr Nowojski" <[hidden email]> wrote:

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <[hidden email]> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.

 

 

 

 

[mingleizhang-2]

I found a website: https://issues.apache.org/jira/browse/FLINK-4360 implemented this before.

Rice.

At 2018-02-05 17:56:49, "Piotr Nowojski" <[hidden email]> wrote:

It seems so - but I’m saying this only basing on a annotations when this method was added (in the last couple of months). I’m not that much familiar with those code parts.

Piotrek

On 5 Feb 2018, at 10:51, mingleizhang <[hidden email]> wrote:

 Makes sense to me now. Is it a new design at FLIP6 ?

Rice.

At 2018-02-05 17:49:05, "Piotr Nowojski" <[hidden email]> wrote:

I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).

Piotrek

On 5 Feb 2018, at 10:44, mingleizhang <[hidden email]> wrote:

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?

Rice.

At 2018-02-05 17:30:32, "Piotr Nowojski" <[hidden email]> wrote:

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <[hidden email]> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.

 

 

 

 

 

[Piotr Nowojski]

Yes, but this issue is still a part of the FLIP-6 work.

Piotrek

On 5 Feb 2018, at 11:01, mingleizhang <[hidden email]> wrote:

I found a website: https://issues.apache.org/jira/browse/FLINK-4360 implemented this before.

Rice.

At 2018-02-05 17:56:49, "Piotr Nowojski" <[hidden email]> wrote:

It seems so - but I’m saying this only basing on a annotations when this method was added (in the last couple of months). I’m not that much familiar with those code parts.

Piotrek

On 5 Feb 2018, at 10:51, mingleizhang <[hidden email]> wrote:

 Makes sense to me now. Is it a new design at FLIP6 ?

Rice.

At 2018-02-05 17:49:05, "Piotr Nowojski" <[hidden email]> wrote:

I might be wrong but I think it is other way around and the naming of this method is correct - it does exactly what it says. TaskManager comes with some predefined task slots and it is the one that is offering them to a JobManager. JobManager can use those slots offers to (later!) schedule tasks. (#offerSlotsToJobManager() is being called during TaskManager initialisation).

Piotrek

On 5 Feb 2018, at 10:44, mingleizhang <[hidden email]> wrote:

Yes. Thanks Piotrek. Of course. So, TaskExecutor#offerSlotsToJobManager sounds confuse to me. It might be better to rename it to requestSlotsFromJobManager. I dont know whether it is sounds OKay for that. I just feel like offerSlotToJobManager sounds strange.. What do you think of this ?

Rice.

At 2018-02-05 17:30:32, "Piotr Nowojski" <[hidden email]> wrote:

org.apache.flink.runtime.jobmaster.JobMaster#offerSlots is a receiver side of an RPC call that is being initiated on the sender side: org.apache.flink.runtime.taskexecutor.TaskExecutor#offerSlotsToJobManager.

In other words, JobMasterGateway.offerSlots is called by a TaskManager and it is a way how TaskManager is advertising his slots to a JobManager.

Piotrek

On 5 Feb 2018, at 08:38, mingleizhang <[hidden email]> wrote:

I find some codes in flink does not make sense to me. Like in some classes below

JobMasterGateway.java has a offerSlots method which means Offers the given slots to the job manager. I was wondering why a jobmanager running should need slots ?

TaskExecutor.java has a offerSlotsToJobManager method which means offer slots to jobmanager.

Above both are confuse me. I just know that Task running needs slots which support by a taskManager. Does anyone let me why what does jobmanager needs slots mean ?

Thanks in advance.

Rice.