Setting number of TaskManagers

[Foster, Craig]

Is there a way to set the number of TaskManagers using a configuration file or environment variable? I'm looking at the docs for it and it says you can set slots but not the number of TMs.

 

Thanks,

Craig

[Greg Hogan]

The number of TaskManagers will be equal to the number of entries in the conf/slaves file.

On Wed, Aug 24, 2016 at 3:04 PM, Foster, Craig <[hidden email]> wrote:

Is there a way to set the number of TaskManagers using a configuration file or environment variable? I'm looking at the docs for it and it says you can set slots but not the number of TMs.

 

Thanks,

Craig

[Foster, Craig]

Oh, sorry, I didn't specify I was using YARN and don't necessarily want to specify with the command line option.

 

From: Greg Hogan <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, August 24, 2016 at 12:07 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Setting number of TaskManagers

 

The number of TaskManagers will be equal to the number of entries in the conf/slaves file.

 

On Wed, Aug 24, 2016 at 3:04 PM, Foster, Craig <[hidden email]> wrote:

Is there a way to set the number of TaskManagers using a configuration file or environment variable? I'm looking at the docs for it and it says you can set slots but not the number of TMs.

 

Thanks,

Craig

 

[rmetzger0]

Hi Craig,

For the YARN session, you have to pass the the number of taskManagers using the -n argument.

if you need to use a n environment variable, you can create a custom script calling the yarn-session.sh script and passing the value of the env variable to the script.

Regards,

Robert

On Wed, Aug 24, 2016 at 9:11 PM, Foster, Craig <[hidden email]> wrote:

Oh, sorry, I didn't specify I was using YARN and don't necessarily want to specify with the command line option.

 

From: Greg Hogan <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, August 24, 2016 at 12:07 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Setting number of TaskManagers

 

The number of TaskManagers will be equal to the number of entries in the conf/slaves file.

 

On Wed, Aug 24, 2016 at 3:04 PM, Foster, Craig <[hidden email]> wrote:

Is there a way to set the number of TaskManagers using a configuration file or environment variable? I'm looking at the docs for it and it says you can set slots but not the number of TMs.

 

Thanks,

Craig