on YARN question
Locked
 
|
Hello, could you please tell me how to solve the problem that when I use yarn-session.sh, the card will not run when it reaches the following place? Hadoop2.7.2 flink 1.10.0 have： flink/lib/ flink-shaded-hadoop-2-uber-2.7.5-10.0.jar [root@hadoop131 bin]# ./yarn-session.sh -n 2 -s 2 -jm 1024 -tm 1024 -nm test -d 2020-04-10 11:00:35,434 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, hadoop131 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1568m 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2020-04-10 11:00:35,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region 2020-04-10 11:00:35,553 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn properties file under /tmp/.yarn-properties-root. 2020-04-10 11:00:36,141 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2020-04-10 11:00:36,323 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE) 2020-04-10 11:00:36,509 INFO org.apache.flink.runtime.security.modules.JaasModule - Jaas file will be created as /tmp/jaas-9182197754252132172.conf. 2020-04-10 11:00:36,554 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli - The configuration directory ('/opt/module/flink-1.10.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2020-04-10 11:00:36,653 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop132/192.168.15.132:8032 2020-04-10 11:00:36,903 INFO org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils - The derived from fraction jvm overhead memory (156.800mb (164416719 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead 2020-04-10 11:00:37,048 WARN org.apache.flink.yarn.YarnClusterDescriptor - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN. 2020-04-10 11:00:37,109 INFO org.apache.flink.yarn.YarnClusterDescriptor - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1568, slotsPerTaskManager=1} 2020-04-10 11:00:50,693 INFO org.apache.flink.yarn.YarnClusterDescriptor - Submitting application master application_1586487382351_0001 2020-04-10 11:00:51,093 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1586487382351_0001 2020-04-10 11:00:51,093 INFO org.apache.flink.yarn.YarnClusterDescriptor - Waiting for the cluster to be allocated 2020-04-10 11:00:51,096 INFO org.apache.flink.yarn.YarnClusterDescriptor - Deploying cluster, current state ACCEPTED 2020-04-10 11:01:04,140 INFO org.apache.flink.yarn.YarnClusterDescriptor - YARN application has been deployed successfully. 2020-04-10 11:01:04,141 INFO org.apache.flink.yarn.YarnClusterDescriptor - Found Web Interface hadoop133:40677 of application 'application_1586487382351_0001'. JobManager Web Interface: http://hadoop133:40677 |
|
|
I am not a Flink expert. Just out of curiosity, I am seeing “YARN application has been deployed successfully“ Does it not mean it’s working properly? Best, Ethan On Apr 9, 2020, at 23:01, 罗杰 <[hidden email]> wrote:
|
|
|
Do you mean to run it in detach mode? If so, you could add "-d".
Best, Yangze Guo On Fri, Apr 10, 2020 at 1:05 PM Ethan Li <[hidden email]> wrote: > > I am not a Flink expert. Just out of curiosity, > > I am seeing > > “YARN application has been deployed successfully“ > > Does it not mean it’s working properly? > > > Best, > Ethan > > On Apr 9, 2020, at 23:01, 罗杰 <[hidden email]> wrote: > > > Hello, could you please tell me how to solve the problem that when I use yarn-session.sh, the card will not run when it reaches the following place? > Hadoop2.7.2 flink 1.10.0 > have： flink/lib/ flink-shaded-hadoop-2-uber-2.7.5-10.0.jar > [root@hadoop131 bin]# ./yarn-session.sh -n 2 -s 2 -jm 1024 -tm 1024 -nm test -d > 2020-04-10 11:00:35,434 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, hadoop131 > 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 > 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m > 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1568m > 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1 > 2020-04-10 11:00:35,437 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 > 2020-04-10 11:00:35,438 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region > 2020-04-10 11:00:35,553 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn properties file under /tmp/.yarn-properties-root. > 2020-04-10 11:00:36,141 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable > 2020-04-10 11:00:36,323 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE) > 2020-04-10 11:00:36,509 INFO org.apache.flink.runtime.security.modules.JaasModule - Jaas file will be created as /tmp/jaas-9182197754252132172.conf. > 2020-04-10 11:00:36,554 WARN org.apache.flink.yarn.cli.FlinkYarnSessionCli - The configuration directory ('/opt/module/flink-1.10.0/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. > 2020-04-10 11:00:36,653 INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at hadoop132/192.168.15.132:8032 > 2020-04-10 11:00:36,903 INFO org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils - The derived from fraction jvm overhead memory (156.800mb (164416719 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead > 2020-04-10 11:00:37,048 WARN org.apache.flink.yarn.YarnClusterDescriptor - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN. > 2020-04-10 11:00:37,109 INFO org.apache.flink.yarn.YarnClusterDescriptor - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1568, slotsPerTaskManager=1} > 2020-04-10 11:00:50,693 INFO org.apache.flink.yarn.YarnClusterDescriptor - Submitting application master application_1586487382351_0001 > 2020-04-10 11:00:51,093 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1586487382351_0001 > 2020-04-10 11:00:51,093 INFO org.apache.flink.yarn.YarnClusterDescriptor - Waiting for the cluster to be allocated > 2020-04-10 11:00:51,096 INFO org.apache.flink.yarn.YarnClusterDescriptor - Deploying cluster, current state ACCEPTED > 2020-04-10 11:01:04,140 INFO org.apache.flink.yarn.YarnClusterDescriptor - YARN application has been deployed successfully. > 2020-04-10 11:01:04,141 INFO org.apache.flink.yarn.YarnClusterDescriptor - Found Web Interface hadoop133:40677 of application 'application_1586487382351_0001'. > JobManager Web Interface: http://hadoop133:40677 > |
|
|
Hi, I think you need to remove "-n 2" from your start up command. The "-n" option has been removed and no longer take effect. Flink will stop parsing the command line options once it see an un-recognized token. That means all the options after "-n 2" are ignored, including "-d". As a result, you're running the session cluster in attached mode, and the client will not exit until the session is shutdown. Thank you~ Xintong Song On Fri, Apr 10, 2020 at 1:10 PM Yangze Guo <[hidden email]> wrote: Do you mean to run it in detach mode? If so, you could add "-d". |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |