YARN : Different cutoff for job and task managers

[Gwenhael Pasquiers]

Hello,

 

In a setup where we allocate most of the memory to rocksdb (off-heap) we ha= ve an important cutoff.

 

Our issue is that the same cutoff applies to both task and job managers : the heap size of the job manager then becomes too low.

 

Is there a way to apply different cutoffs to job and task managers ?

 

Regards,

[Yang Wang]

Hi Gwenhael,

I'm afraid that we could not set different cut-off to jobmanager and taskmanager. You could set

the jvm args manually to work around. For example, 'env.java.opts.jobmanager=-Xms3072m -Xmx3072m'.

In most jvm implementation, the rightmost Xmx Xms will take effect. So i think it should work.

Please have a try.

Best,

Yang

Gwenhael Pasquiers <[hidden email]> 于2019年11月19日周二 下午10:56写道：

Hello,

 

In a setup where we allocate most of the memory to rocksdb (off-heap) we ha= ve an important cutoff.

 

Our issue is that the same cutoff applies to both task and job managers : the heap size of the job manager then becomes too low.

 

Is there a way to apply different cutoffs to job and task managers ?

 

Regards,

[Gwenhael Pasquiers]

I see, good idea,  I’ll try that and tell you the result.

 

Thanks,

 

From: Yang Wang <[hidden email]>
Sent: mercredi 20 novembre 2019 03:52
To: Gwenhael Pasquiers <[hidden email]>
Cc: [hidden email]
Subject: Re: YARN : Different cutoff for job and task managers

 

Hi Gwenhael,

 

I'm afraid that we could not set different cut-off to jobmanager and taskmanager. You could set

the jvm args manually to work around. For example, 'env.java.opts.jobmanager=-Xms3072m -Xmx3072m'.

In most jvm implementation, the rightmost Xmx Xms will take effect. So i think it should work.

Please have a try.

 

 

Best,

Yang

 

Gwenhael Pasquiers <[hidden email]> 于2019年11月19日周二 下午10:56写道：

Hello,

 

In a setup where we allocate most of the memory to rocksdb (off-heap) we ha= ve an important cutoff.

 

Our issue is that the same cutoff applies to both task and job managers : the heap size of the job manager then becomes too low.

 

Is there a way to apply different cutoffs to job and task managers ?

 

Regards,

[Gwenhael Pasquiers]

This works; I had some issues modifying my scripts, but it’s OK and I could confirm by JMX that env.java.opts.jobmanager had priority over the “normal” heap size (calculated from cutoff).

 

Thanks ! 😊

 

From: Yang Wang <[hidden email]>
Sent: mercredi 20 novembre 2019 03:52
To: Gwenhael Pasquiers <[hidden email]>
Cc: [hidden email]
Subject: Re: YARN : Different cutoff for job and task managers

 

Hi Gwenhael,

 

I'm afraid that we could not set different cut-off to jobmanager and taskmanager. You could set

the jvm args manually to work around. For example, 'env.java.opts.jobmanager=-Xms3072m -Xmx3072m'.

In most jvm implementation, the rightmost Xmx Xms will take effect. So i think it should work.

Please have a try.

 

 

Best,

Yang

 

Gwenhael Pasquiers <[hidden email]> 于2019年11月19日周二 下午10:56写道：

Hello,

 

In a setup where we allocate most of the memory to rocksdb (off-heap) we ha= ve an important cutoff.

 

Our issue is that the same cutoff applies to both task and job managers : the heap size of the job manager then becomes too low.

 

Is there a way to apply different cutoffs to job and task managers ?

 

Regards,