Extending Flink Slots when running on Yarn

[Julio Biason]

Hi,

I'm wodering, is there a way to make Flink understand that I added a new worker machine on yarn to increase the tasks slots?

Right now, we are just exploring how to run Flink on Yarn and, so far, we managed to create a small Hadoop/Yarn cluster (3 machines, 1 master and 2 workers) and start Flink on it.

But say I want to extend this cluster to add more machines; in this case, simply adding more machines to the Hadoop cluster will work?

--

[Gary Yao-2]

Hi Julio,

When you start the Flink YARN session, you have to specify the number of

TaskManagers and number of slots per TaskManager. There is currently no

officially supported way to add more TaskManagers to a long running YARN

session. We are aware of this limitation, and there are ongoing development

efforts to make the Flink Cluster dynamically allocate new YARN containers when

needed (FLIP-6) [1]. We are aiming at shipping this feature with the Flink 1.5

release [2]. As a workaround, you can restore your jobs from savepoints on a

bigger cluster.

In a standalone cluster it is possible to add TaskManagers to a running cluster

[3].

Best,

Gary

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077

[2] https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html

[3] https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/cluster_setup.html#adding-jobmanagertaskmanager-instances-to-a-cluster

On Thu, Feb 1, 2018 at 5:09 PM, Julio Biason <[hidden email]> wrote:

Hi,

I'm wodering, is there a way to make Flink understand that I added a new worker machine on yarn to increase the tasks slots?

Right now, we are just exploring how to run Flink on Yarn and, so far, we managed to create a small Hadoop/Yarn cluster (3 machines, 1 master and 2 workers) and start Flink on it.

But say I want to extend this cluster to add more machines; in this case, simply adding more machines to the Hadoop cluster will work?

--