Internal buffers supervision and yarn vCPUs

[Gwenhael Pasquiers]

Hi,

 

I’ve got two more questions on different topic…

 

First one :

Is there a way to monitor the buffers status. In order to  find bottleneck in our application we though it could be usefull to be able to have a look at the different exchange buffers’ status. To know if they are full (or as an example if a mapper had to wait before being able to push it’s data into the buffer). That way we can know where the bottleneck is.

 

Second one :

On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc” argument that allowed to specify the number of vCPU per task manager. We cannot find it anymore. Was it removed ? Is there another way to set the number of vCPU. Or did it became useless ?

 

Thanks in advance.

 

Gwen’

[rmetzger0]

Hi Gwen,

let me answer the second question: There is a JIRA to reintroduce the configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213. I will try to get a fix for this into the 1.0 release.

I think I removed back then because users were unable to define the number of vcores independently of the number of slots ... and too many users were running into issues with the yarn scheduler (containers were not started because there were no CPU resources available anymore).

On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <[hidden email]> wrote:

Hi,

 

I’ve got two more questions on different topic…

 

First one :

Is there a way to monitor the buffers status. In order to  find bottleneck in our application we though it could be usefull to be able to have a look at the different exchange buffers’ status. To know if they are full (or as an example if a mapper had to wait before being able to push it’s data into the buffer). That way we can know where the bottleneck is.

 

Second one :

On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc” argument that allowed to specify the number of vCPU per task manager. We cannot find it anymore. Was it removed ? Is there another way to set the number of vCPU. Or did it became useless ?

 

Thanks in advance.

 

Gwen’

[Stephan Ewen]

Concerning the first question:

What you are looking for is backpressure monitoring. If a task cannot push its data to the next task, it is backpressured.

This pull request adds a first version of backpressure monitoring: https://github.com/apache/flink/pull/1578

We will try and get it merged soon!

On Thu, Feb 4, 2016 at 11:03 AM, Robert Metzger <[hidden email]> wrote:

Hi Gwen,

let me answer the second question: There is a JIRA to reintroduce the configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213. I will try to get a fix for this into the 1.0 release.

I think I removed back then because users were unable to define the number of vcores independently of the number of slots ... and too many users were running into issues with the yarn scheduler (containers were not started because there were no CPU resources available anymore).

On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <[hidden email]> wrote:

Hi,

 

I’ve got two more questions on different topic…

 

First one :

Is there a way to monitor the buffers status. In order to  find bottleneck in our application we though it could be usefull to be able to have a look at the different exchange buffers’ status. To know if they are full (or as an example if a mapper had to wait before being able to push it’s data into the buffer). That way we can know where the bottleneck is.

 

Second one :

On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc” argument that allowed to specify the number of vCPU per task manager. We cannot find it anymore. Was it removed ? Is there another way to set the number of vCPU. Or did it became useless ?

 

Thanks in advance.

 

Gwen’

[Gwenhael Pasquiers]

Ok thanks !

 

All that’s left is to wait then.

 

B.R.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Stephan Ewen
Sent: jeudi 4 février 2016 11:19
To: [hidden email]
Subject: Re: Internal buffers supervision and yarn vCPUs

 

Concerning the first question:

 

What you are looking for is backpressure monitoring. If a task cannot push its data to the next task, it is backpressured.

 

This pull request adds a first version of backpressure monitoring: https://github.com/apache/flink/pull/1578

 

We will try and get it merged soon!

 

 

On Thu, Feb 4, 2016 at 11:03 AM, Robert Metzger <[hidden email]> wrote:

Hi Gwen,

 

let me answer the second question: There is a JIRA to reintroduce the configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213. I will try to get a fix for this into the 1.0 release.

 

I think I removed back then because users were unable to define the number of vcores independently of the number of slots ... and too many users were running into issues with the yarn scheduler (containers were not started because there were no CPU resources available anymore).

 

 

 

On Thu, Feb 4, 2016 at 10:56 AM, Gwenhael Pasquiers <[hidden email]> wrote:

Hi,

 

I’ve got two more questions on different topic…

 

First one :

Is there a way to monitor the buffers status. In order to  find bottleneck in our application we though it could be usefull to be able to have a look at the different exchange buffers’ status. To know if they are full (or as an example if a mapper had to wait before being able to push it’s data into the buffer). That way we can know where the bottleneck is.

 

Second one :

On type of resources on yarn is vCPU. In flink 0.8 there was a “-tmc” argument that allowed to specify the number of vCPU per task manager. We cannot find it anymore. Was it removed ? Is there another way to set the number of vCPU. Or did it became useless ?

 

Thanks in advance.

 

Gwen’

 

 

[Ufuk Celebi]

> On 04 Feb 2016, at 12:02, Gwenhael Pasquiers <[hidden email]> wrote:
>
> Ok thanks !
>  
> All that’s left is to wait then.

If you have spare time and are working with the current snapshot version, it would be great to get some feedback on the pull request. :-)

– Ufuk