How to use all available task managers

[Saiph Kappa]

Hi,

I  am running a flink stream application on a cluster with 6 slaves/task managers. I have set in flink-conf.yaml of every machine "parallelization.degree.default: 6". However, when I run my application it just uses one task slot and not all of them. Am I missing something?

Thanks.

[Till Rohrmann]

Hi Saiph,

I think the configuration value should be parallelism.default: 6. That will execute jobs which have not parallelism defined with a DOP of 6.

Cheers,
Till

​

On Wed, Feb 24, 2016 at 1:43 AM, Saiph Kappa <[hidden email]> wrote:

Hi,

I  am running a flink stream application on a cluster with 6 slaves/task managers. I have set in flink-conf.yaml of every machine "parallelization.degree.default: 6". However, when I run my application it just uses one task slot and not all of them. Am I missing something?

Thanks.

[Saiph Kappa]

I set "parallelism.default: 6" on flink-conf.yaml of all 6 machines, and still, my job only uses 1 task manager. Why?

On Wed, Feb 24, 2016 at 8:31 AM, Till Rohrmann <[hidden email]> wrote:

Hi Saiph,

I think the configuration value should be parallelism.default: 6. That will execute jobs which have not parallelism defined with a DOP of 6.

Cheers,
Till

​

On Wed, Feb 24, 2016 at 1:43 AM, Saiph Kappa <[hidden email]> wrote:

Hi,

I  am running a flink stream application on a cluster with 6 slaves/task managers. I have set in flink-conf.yaml of every machine "parallelization.degree.default: 6". However, when I run my application it just uses one task slot and not all of them. Am I missing something?

Thanks.

[Ufuk Celebi]

You can use the environment to set it the job parallelism to 6 e.g.
env.setParallelism(6).

Setting this will override the default behaviour. Maybe that's why the
default parallelism is not working... you might have it set to 1
already?

On Wed, Feb 24, 2016 at 3:41 PM, Saiph Kappa <[hidden email]> wrote:

> I set "parallelism.default: 6" on flink-conf.yaml of all 6 machines, and
> still, my job only uses 1 task manager. Why?
>
>
> On Wed, Feb 24, 2016 at 8:31 AM, Till Rohrmann <[hidden email]> wrote:
>>
>> Hi Saiph,
>>
>> I think the configuration value should be parallelism.default: 6. That
>> will execute jobs which have not parallelism defined with a DOP of 6.
>>
>> Cheers,
>> Till
>>
>>
>> On Wed, Feb 24, 2016 at 1:43 AM, Saiph Kappa <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>>
>>> I  am running a flink stream application on a cluster with 6 slaves/task
>>> managers. I have set in flink-conf.yaml of every machine
>>> "parallelization.degree.default: 6". However, when I run my application it
>>> just uses one task slot and not all of them. Am I missing something?
>>>
>>> Thanks.
>>
>>
>

[Saiph Kappa]

Thanks! It worked now :-)

On Wed, Feb 24, 2016 at 2:48 PM, Ufuk Celebi <[hidden email]> wrote:

You can use the environment to set it the job parallelism to 6 e.g.
env.setParallelism(6).

Setting this will override the default behaviour. Maybe that's why the
default parallelism is not working... you might have it set to 1
already?

On Wed, Feb 24, 2016 at 3:41 PM, Saiph Kappa <[hidden email]> wrote:
> I set "parallelism.default: 6" on flink-conf.yaml of all 6 machines, and
> still, my job only uses 1 task manager. Why?
>
>
> On Wed, Feb 24, 2016 at 8:31 AM, Till Rohrmann <[hidden email]> wrote:
>>
>> Hi Saiph,
>>
>> I think the configuration value should be parallelism.default: 6. That
>> will execute jobs which have not parallelism defined with a DOP of 6.
>>
>> Cheers,
>> Till
>>
>>
>> On Wed, Feb 24, 2016 at 1:43 AM, Saiph Kappa <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>>
>>> I  am running a flink stream application on a cluster with 6 slaves/task
>>> managers. I have set in flink-conf.yaml of every machine
>>> "parallelization.degree.default: 6". However, when I run my application it
>>> just uses one task slot and not all of them. Am I missing something?
>>>
>>> Thanks.
>>
>>
>

[Matthias J. Sax-2]

Could it be, that you would need to edit client local flink-conf.yaml
instead of the TaskManager config files? (In case, you do not want to
specify parallelism via env.setParallelism(int);)

-Matthias

On 02/24/2016 04:19 PM, Saiph Kappa wrote:

> Thanks! It worked now :-)
>
> On Wed, Feb 24, 2016 at 2:48 PM, Ufuk Celebi <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     You can use the environment to set it the job parallelism to 6 e.g.
>     env.setParallelism(6).
>
>     Setting this will override the default behaviour. Maybe that's why the
>     default parallelism is not working... you might have it set to 1
>     already?
>
>     On Wed, Feb 24, 2016 at 3:41 PM, Saiph Kappa <[hidden email]
>     <mailto:[hidden email]>> wrote:
>     > I set "parallelism.default: 6" on flink-conf.yaml of all 6
>     machines, and
>     > still, my job only uses 1 task manager. Why?
>     >
>     >
>     > On Wed, Feb 24, 2016 at 8:31 AM, Till Rohrmann
>     <[hidden email] <mailto:[hidden email]>> wrote:
>     >>
>     >> Hi Saiph,
>     >>
>     >> I think the configuration value should be parallelism.default: 6.
>     That
>     >> will execute jobs which have not parallelism defined with a DOP of 6.
>     >>
>     >> Cheers,
>     >> Till
>     >>
>     >>
>     >> On Wed, Feb 24, 2016 at 1:43 AM, Saiph Kappa
>     <[hidden email] <mailto:[hidden email]>>
>     >> wrote:
>     >>>
>     >>> Hi,
>     >>>
>     >>> I  am running a flink stream application on a cluster with 6
>     slaves/task
>     >>> managers. I have set in flink-conf.yaml of every machine
>     >>> "parallelization.degree.default: 6". However, when I run my
>     application it
>     >>> just uses one task slot and not all of them. Am I missing something?
>     >>>
>     >>> Thanks.
>     >>
>     >>
>     >
>
>