Hi Deep,
Flink has dropped support for specifying the number of TMs via -n since the introduction of Flip-6. Since then, Flink will automatically start TMs depending on the required resources. Hence, there is no need to specify the -n parameter anymore. Instead, you should specify the parallelism with which you would like to run your job via the -p option.
Since Flink 1.11.0 there is the option slotmanager.number-of-slots.max to limit the upper limit of slots a cluster is allowed to allocate [1].
Cheers,
Till
On Mon, Jan 4, 2021 at 8:33 AM DEEP NARAYAN Singh <
[hidden email]> wrote:
Hi Guys,
I’m struggling while initiating the task manager with flink 1.11.0 in AWS
EMR but with older versions it is not. Let me put the full context here.
*When using Flink 1.9.1 and EMR 5.29.0*
To create a long running session, we used the below command.
*sudo flink-yarn-session -n <Number of TM> -s <Number of slot> -jm <memory>
-tm <memory> -d*
and followed by below command to run the final job.
*flink run -m yarn-cluster -yid <flink sessionId> -yn <Number of TM> -ys
<Number of slot> -yjm <memory> -ytm <memory> -c <ClassName> <Jar Path>*
and if “n” is 6 then it is used to create 6 task managers to start the job,
so whatever “n” is configured the result was that number of TM the job is
being started.
But Now when we scaled up with the configuration (*i.e. Flink 1.11.0 and
EMR 6.1.0*) we are unable to achieve the desired values for TM.
Please find the session Ids of new configuration,
*sudo flink-yarn-session -Djobmanager.memory.process.size=<Memory in GB>
-Dtaskmanager.memory.process.size=<Memory in GB> -n <no of TM> -s <No of
slot/core> -d*
And the final Job command
*flink run -m yarn-cluster -yid <Flink sessionId> -c <ClassName> <Jar Path>*
I have tried a lot of combinations, but nothing worked out so far. I
request your help in this regard as the plan to have this configuration in
*PRODUCTION* soon.
Thanks in advance.
Regards,
-Deep
Locked
