Flink configuration at runtime

[amran dean]

Is it possible to configure certain settings at runtime, on a per-job basis rather than globally within flink-conf.yaml?

For example, I have a job where it's desirable to retain a large number of checkpoints via 

state.checkpoints.num-retained.

The checkpoints are cheap, and it's low cost. For other jobs, I don't want such a large number.

[Zhu Zhu]

Hi Amran,

Some configs, including "state.checkpoints.num-retained", are cluster configs that always apply to the entire Flink cluster.

An alternative is to use per-job mode if you are running Flink jobs on k8s/docker or yarn. Thus to create a Flink cluster for a single job.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/docker.html#flink-job-cluster

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn

Thanks,

Zhu Zhu 

amran dean <[hidden email]> 于2019年11月19日周二 上午5:53写道：

Is it possible to configure certain settings at runtime, on a per-job basis rather than globally within flink-conf.yaml?

For example, I have a job where it's desirable to retain a large number of checkpoints via 

state.checkpoints.num-retained.

The checkpoints are cheap, and it's low cost. For other jobs, I don't want such a large number.

[vino yang]

Hi Amran,

Change the config option at runtime? No, Flink does not support this feature currently.

However, for Flink on Yarn job cluster mode, you can specify different config options for different jobs via program or flink-conf.yaml(copy a new flink binary package then change config file).

Best,

Vino

amran dean <[hidden email]> 于2019年11月19日周二 上午5:53写道：

Is it possible to configure certain settings at runtime, on a per-job basis rather than globally within flink-conf.yaml?

For example, I have a job where it's desirable to retain a large number of checkpoints via 

state.checkpoints.num-retained.

The checkpoints are cheap, and it's low cost. For other jobs, I don't want such a large number.

[rmetzger0]

Hi Amran,

thanks a lot for your message.

I think this is a valid feature request. I've created a JIRA ticket to track it: https://issues.apache.org/jira/browse/FLINK-14856 (this does not mean this gets addressed immediately. However, there are currently quite some improvements to the configuration system in Flink, as part of FLIP-59 and FLIP-81)

Best,

Robert

On Tue, Nov 19, 2019 at 4:09 AM vino yang <[hidden email]> wrote:

Hi Amran,

Change the config option at runtime? No, Flink does not support this feature currently.

However, for Flink on Yarn job cluster mode, you can specify different config options for different jobs via program or flink-conf.yaml(copy a new flink binary package then change config file).

Best,

Vino

amran dean <[hidden email]> 于2019年11月19日周二 上午5:53写道：

Is it possible to configure certain settings at runtime, on a per-job basis rather than globally within flink-conf.yaml?

For example, I have a job where it's desirable to retain a large number of checkpoints via 

state.checkpoints.num-retained.

The checkpoints are cheap, and it's low cost. For other jobs, I don't want such a large number.