Flink on Yarn resource arrangement

[qq]

Hi all,

   Could you list details how  Flink job on Yarn resources managed ?

  I used command “-p 20 -yn 5 -ys 3 -yjm 2048m -ytm 2048m” to run flink job. I got
containers vcores
    8      22
Task Managers 7 Total Task Slots 21


I used command “-p 20 -yn 7 -ys 4 -yjm 2048m -ytm 2048m” to run flink job, I got
containers vcores
   6            21
Total Task Slots 20 Task Managers 5

Could you help give the exactly resources formula ? Thanks very much.



Alex Fu
2019/11/14

[vino yang]

Hi Alex,

Which Flink version are you using?

AFAIK, since Flink 1.8+, the config option: "-yn" for Flink on YARN job cluster mode does not take effect(always 1 and would be overridden).

So, the config option "-ys" and "-p" will decide the number of TM.

The first example: -p(20)/-ys(3) should be advanced to 7 to match the requirement of parallelism. So total slots should be 7 * 3 = 21.

The second example: -p(20)/-ys(4) = 5. So total slots should be 5 * 4 = 20.

Best,

Vino

qq <[hidden email]> 于2019年11月14日周四 下午3:26写道：

Hi all,

   Could you list details how  Flink job on Yarn resources managed ?

  I used command “-p 20 -yn 5 -ys 3 -yjm 2048m -ytm 2048m” to run flink job. I got
containers vcores
    8         22
Task Managers 7 Total Task Slots 21


I used command “-p 20 -yn 7 -ys 4 -yjm 2048m -ytm 2048m” to run flink job, I got
containers vcores
   6                21
Total Task Slots 20 Task Managers 5

Could you help give the exactly resources formula ? Thanks very much.



Alex Fu
2019/11/14