Flink on YARN: long-running session vs. one-off jobs

Posted by stefanobaghino on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-on-YARN-long-running-session-vs-one-off-jobs-tp5321.html

Hello everybody,

I'm currently studying how the Flink/YARN integration works. Right now I'm a little confused about the practical difference in having a long-running session on which one deploys several jobs or deploying these jobs individually.

My intuition (which may not be correct) is that in the former ase a pool of resources is allocated to the session and then the usage of these is handled by the Flink JobManager, while in the latter case the resource allocation is handled directly by YARN on a per-job basis. Am I right?

If what I said is (more or less) right and apart from security concerns (which have been discussed in a previous thread), are there any further practical differences between having a long-running session or letting YARN handle jobs?

Thank you in advance!

--