degree of Parallelism

[Ahmed Nader]

Hi,

How can i setParallelism in a generic way that provide better performance on any device not only mine? In this case is it better to be set to a certain value or if i just didn't set it to any value does flink take care of that generically and provide better execution performance? 

Thanks.

[Till Rohrmann]

Hi Ahmed,

if you don't set the parallelism in your program then depending on how you execute your program different parallelisms will be used. If you execute it in your IDE, then the number of cores will be used as parallelism. If you submit it to a cluster without specifying the parallelism via the cli, then the default parallelism which you've specified in flink-conf.yaml will be used. If you use the -p option of the cli, then this value will be used as the parallelism.

Cheers,

Till

On Fri, Mar 18, 2016 at 1:13 PM, Ahmed Nader <[hidden email]> wrote:

Hi,

How can i setParallelism in a generic way that provide better performance on any device not only mine? In this case is it better to be set to a certain value or if i just didn't set it to any value does flink take care of that generically and provide better execution performance? 

Thanks.

[Fabian Hueske-2]

Hi,

did find the documentation for configuring the parallelism [1]?

It explains how to set the parallelism on different levels: Cluster, Job, Task.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/common/index.html#parallel-execution

2016-03-18 13:34 GMT+01:00 Till Rohrmann <[hidden email]>:

Hi Ahmed,

if you don't set the parallelism in your program then depending on how you execute your program different parallelisms will be used. If you execute it in your IDE, then the number of cores will be used as parallelism. If you submit it to a cluster without specifying the parallelism via the cli, then the default parallelism which you've specified in flink-conf.yaml will be used. If you use the -p option of the cli, then this value will be used as the parallelism.

Cheers,

Till

On Fri, Mar 18, 2016 at 1:13 PM, Ahmed Nader <[hidden email]> wrote:

Hi,

How can i setParallelism in a generic way that provide better performance on any device not only mine? In this case is it better to be set to a certain value or if i just didn't set it to any value does flink take care of that generically and provide better execution performance? 

Thanks.

[Ahmed Nader]

Thanks so much Till and Fabian. So if i were to set it, how can i know the best degree to be used for my application, is it the number of cores or what should i set it to?

Thanks

On 18 March 2016 at 13:36, Fabian Hueske <[hidden email]> wrote:

Hi,

did find the documentation for configuring the parallelism [1]?

It explains how to set the parallelism on different levels: Cluster, Job, Task.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/common/index.html#parallel-execution

2016-03-18 13:34 GMT+01:00 Till Rohrmann <[hidden email]>:

Hi Ahmed,

if you don't set the parallelism in your program then depending on how you execute your program different parallelisms will be used. If you execute it in your IDE, then the number of cores will be used as parallelism. If you submit it to a cluster without specifying the parallelism via the cli, then the default parallelism which you've specified in flink-conf.yaml will be used. If you use the -p option of the cli, then this value will be used as the parallelism.

Cheers,

Till

On Fri, Mar 18, 2016 at 1:13 PM, Ahmed Nader <[hidden email]> wrote:

Hi,

How can i setParallelism in a generic way that provide better performance on any device not only mine? In this case is it better to be set to a certain value or if i just didn't set it to any value does flink take care of that generically and provide better execution performance? 

Thanks.