Adjusting number of YARN containers

[Peter Voß]

Hi,

I have started a Flink YARN session using yarn-session.sh and the configuration of number of YARN container seems to be pretty static.

Is it possible to have Flink adjust the number of containers depending on the actual workload. E.g. stop containers that are idle for too long and start them up again when more jobs get submitted?
Or are there any plans to have more dynamic resource management?

Thanks,
—Peter

[rmetzger0]

Hi,

Currently, Flink does not support automatic scaling of the YARN containers.

There are certainly plans to add this feature: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/add-some-new-api-to-the-scheduler-in-the-job-manager-tp7153p7448.html

Adding an API for manually starting and stopping containers is something we can do very easily, something dynamic is a bit harder.

The feature is something I was always interested to work on, but nobody really asked for it. But it seems that more and more people are asking for it.

If this is a requirement for you to use Flink, I'm certainly willing to add the feature into Flink.

Best,

Robert

On Wed, Sep 9, 2015 at 11:45 AM, Peter Voß <[hidden email]> wrote:

Hi,

I have started a Flink YARN session using yarn-session.sh and the configuration of number of YARN container seems to be pretty static.

Is it possible to have Flink adjust the number of containers depending on the actual workload. E.g. stop containers that are idle for too long and start them up again when more jobs get submitted?
Or are there any plans to have more dynamic resource management?

Thanks,
—Peter