(DEPRECATED) Apache Flink User Mailing List archive.

Flink session on yarn

classic Classic list List threaded Threaded
♦
♦
 3 messages Options
nishutayal
Reply | Threaded
Open this post in threaded view
♦
♦
|

Flink session on yarn

nishutayal
Hi,

I am trying to start flink session(v1.3.2) on yarn(v 2.7) on HDInsight cluster.  But it throws following error:

Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)
        at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
        at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)
        at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)
        at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)



I am using HDInsight Cluster(HDP3.6).   According to the documentation, set HADOOP_CONF_DIR and YARN_CONF_DIR as well.

Any inputs will be really helpful. Thanks!

-- 
Thanks & Regards,
Nishu Tayal
Thomas Nadler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Flink session on yarn

Thomas Nadler

Hi,

instead of using the RequestHedgingRMFailoverProxyProvider you could try to use the org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.
You can configure this in the YARN configs => yarn.client.failover-proxy-provider.

Kind regards,
Thomas


On 11/20/2017 01:53 PM, Nishu wrote:
Hi,

I am trying to start flink session(v1.3.2) on yarn(v 2.7) on HDInsight cluster.  But it throws following error:

Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)
        at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
        at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)
        at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)
        at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)



I am using HDInsight Cluster(HDP3.6).   According to the documentation, set HADOOP_CONF_DIR and YARN_CONF_DIR as well.

Any inputs will be really helpful. Thanks!

-- 
Thanks & Regards,
Nishu Tayal

-- 
Thomas Nadler * [hidden email] * +49-176-17654608
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Müller
Amtsgericht München, HRB 135082
Federico D'Ambrosio-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Flink session on yarn

Federico D'Ambrosio-2
In reply to this post by nishutayal
Hi Nishu,

did you compile Flink from sources as recommended here [1]?

Regards,
Federico


2017-11-20 13:53 GMT+01:00 Nishu <[hidden email]>:
Hi,

I am trying to start flink session(v1.3.2) on yarn(v 2.7) on HDInsight cluster.  But it throws following error:

Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)
        at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
        at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)
        at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)
        at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)



I am using HDInsight Cluster(HDP3.6).   According to the documentation, set HADOOP_CONF_DIR and YARN_CONF_DIR as well.

Any inputs will be really helpful. Thanks!

-- 
Thanks & Regards,
Nishu Tayal



--
Federico D'Ambrosio
Free forum by Nabble Edit this page