Setting max parallelism via properties

[Padarn Wilson-2]

Hi all,

Sorry for the basic question, but is it possible to set max parallelism using the flink conf file, rather than explicitly in code:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/parallel.html#setting-the-maximum-parallelism

Need this for a PR I am working on for the flink operator: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/pull/425

[Kezhu Wang]

Hi Padarn,

There is a configuration option “pipeline.max-parallelism”.

It is not a cluster wide configuration but client/job/pipeline side configuration which means you should bring this configuration from flink conf file to pipeline generation stage.

If I understand correctly, `flink-on-k8s-operator` uses `flink run`(I found this in `flinkcluster_submit_job_script.go`) to submit job to cluster. This command already cover the bridge work, so I think it should just work in your case.

pipeline-max-parallelism: https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#pipeline-max-parallelism

Best,

Kezhu Wang

On February 28, 2021 at 16:45:03, Padarn Wilson ([hidden email]) wrote:

Hi all,

Sorry for the basic question, but is it possible to set max parallelism using the flink conf file, rather than explicitly in code:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/parallel.html#setting-the-maximum-parallelism

Need this for a PR I am working on for the flink operator: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/pull/425

[Padarn Wilson-2]

Thanks a lot Kezhu, this fits the bill perfectly. 

Thanks,

Padarn

On Sun, Feb 28, 2021 at 7:00 PM Kezhu Wang <[hidden email]> wrote:

Hi Padarn,

There is a configuration option “pipeline.max-parallelism”.

It is not a cluster wide configuration but client/job/pipeline side configuration which means you should bring this configuration from flink conf file to pipeline generation stage.

If I understand correctly, `flink-on-k8s-operator` uses `flink run`(I found this in `flinkcluster_submit_job_script.go`) to submit job to cluster. This command already cover the bridge work, so I think it should just work in your case.

pipeline-max-parallelism: https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#pipeline-max-parallelism

Best,

Kezhu Wang

On February 28, 2021 at 16:45:03, Padarn Wilson ([hidden email]) wrote:

Hi all,

Sorry for the basic question, but is it possible to set max parallelism using the flink conf file, rather than explicitly in code:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/parallel.html#setting-the-maximum-parallelism

Need this for a PR I am working on for the flink operator: https://github.com/GoogleCloudPlatform/flink-on-k8s-operator/pull/425