Managing Flink on Yarn programmatically

[John Sherwood]

I think this is primarily a shortcoming in my ability to grep through Scala efficiently, but are there any resources on how to programmatically spin up & administrate Flink jobs on YARN? The CLI naturally works, but I'd like to build out a service handling the nuances of job management rather than rely on users playing with the YARN or Flink CLIs directly.

[rmetzger0]

Hi,

There is currently no officially supported API for managing Flink on YARN programatically. I think the ideal solution would be something like a YarnExecutionEnvironment.

But you can still do it, by using Flink's internal YARN abstraction.

Check out this YARN test case, that is programmatically controlling a YARN cluster: https://github.com/apache/flink/blob/5c4cb452f06e4333be475b30779eaac2628abf51/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java#L217

On Tue, May 17, 2016 at 6:11 AM, John Sherwood <[hidden email]> wrote:

I think this is primarily a shortcoming in my ability to grep through Scala efficiently, but are there any resources on how to programmatically spin up & administrate Flink jobs on YARN? The CLI naturally works, but I'd like to build out a service handling the nuances of job management rather than rely on users playing with the YARN or Flink CLIs directly.