Flink session on Yarn - ClassNotFoundException
Locked
 
|
Hi,
I’m trying to start a flink (1.3.2) session as explained in the docs (https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/yarn_setup.html#start-a-session), but I keep getting the “ClassNotFoundException” you can see below. I’m running an HDInsight cluster on Azure (HDFS and YARN v2.7.3), and I’m exporting my HADOOP_CONF_DIR before running the yarn-session command. Error while deploying YARN cluster: Couldn't deploy Yarn cluster java.lang.RuntimeException: Couldn't deploy Yarn cluster at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630) at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486) at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483) at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227) at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161) at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94) at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.flink.yarn.AbstractYarnClusterDescriptor.getYarnClient(AbstractYarnClusterDescriptor.java:381) at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:458) at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:441) ... 9 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219) ... 17 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) ... 18 more So far, I’ve check that the yarn class path is correct (it is), I also tried manually linking the “hadoop-yarn-common" jar from my /usr/hdp directories into Flink’s “lib” directory. In that case, I get an “IllegalAccessError”: Exception in thread "main" java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; from class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider at org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init(RequestHedgingRMFailoverProxyProvider.java:75) at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:163) at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94) at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:187) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.flink.yarn.AbstractYarnClusterDescriptor.getYarnClient(AbstractYarnClusterDescriptor.java:381) at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:458) at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:441) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630) at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486) at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483) at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483) Any ideas on what might be going on? This last error makes me think that maybe the error has to do with the HDFS version Flink is compiled against, but I’m not sure. Thanks a lot!
|
Re: Flink session on Yarn - ClassNotFoundException
|
|
Hello,
The ClassNotFoundException indicates that you are using a Flink version that wasn't compiled against hadoop 2.7. Replacing one part of the hadoop dependency will most likely not cut it (or fail mysteriously down the line), so i would suggest to check the downloads page and pick the binary matching your hadoop version. It should be enough to replace the flink-shaded-hadoop2-uber-1.3.2.jar under /lib. On 29.08.2017 11:46, Albert Giménez wrote: Hi,
|
|
|
Hi Chesnay,
Thanks for your reply. I did download the binaries matching my Hadoop version (2.7), that's why I was wondering if the issue had something to do with the exact hadoop version flink is compiled again, or if there might be things that are missing in my environment. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ |
Re: Flink session on Yarn - ClassNotFoundException
|
|
Hi, What is your "hadoop version" output? I'm asking because you said your hadoop distribution is in /usr/hdp so it looks like you're using Hortonworks HDP, just like myself. So, this would be a third party distribution and you'd need to build Flink from source according to this: https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions Federico D'Ambrosio Il 30 ago 2017 13:33, "albert" <[hidden email]> ha scritto: Hi Chesnay, |
|
|
We had the same issue. Get the hdp version, from /usr/hdp/current/hadoop-client/hadoop-common-<version>.jar for example. Then rebuild flink from src: mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=<version> for example: mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.1.0-129 Copy and setup build-target/ to the cluster. Export HADOOP_CONF_DIR and YARN_CONF_DIR according to your env. You should have no problem starting the session. On Wed, Aug 30, 2017 at 6:45 AM, Federico D'Ambrosio <[hidden email]> wrote:
|
|
|
Thanks for the replies :) I managed to get it working following the instructions here, but I found a few issues that I guess were specific to HDInsight, or at least to the HDP version it uses. Trying to summarize: Hadoop version After running “hadoop version”, the result was “2.7.3.2.6.1.3-4”. However, when building I was getting errors that some dependencies from the Hortonworks repo were not found, for instance zookeeper “3.4.6.2.6.1.3-4”. I browsed to the Hortonworks repo to find a suitable version, so I ended up using 2.7.3.2.6.1.31-3 instead. Scala version I also had issues with dependencies if I tried using Scala version 2.11.11, so I compiled agains 2.11.7. So, the maven command I used was this: mvn install -DskipTests -Dscala.version=2.11.7 -Pvendor-repos -Dhadoop.version=2.7.3.2.6.1.31-3 Azure Jars With all that, I still had class not found errors errors when trying to start my Flink session, for instance "java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.HdiAdlFileSystem”. To fix that, I had to find the azure-specific jars I needed to use. For that, I checked which jars Spark was using and copied / symlinked them into Flink’s “lib” directory: /usr/hdp/current/spark2-client/jars/*azure*.jar /usr/lib/hdinsight-datalake/adls2-oauth2-token-provider.jar Guava Jar Finally, for some reason my jobs were failing (the Cassandra driver was complaining about the Guava version being too old, although I had the right version in my assembled jar). I just downloaded the version I needed (in my case, 23.0) and also put that into Flink’s lib directory. I hope it helps other people trying to run Flink on Azure HDInsight :) Kind regards, Albert
|
Re: Flink session on Yarn - ClassNotFoundException
|
|
Hi Albert,
Your thread is really helpful for flink setup on HDInsight cluster. I followed the same and now I hit another error: *Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V at org.apache.hadoop.fs.adl.AdlConfKeys.addDeprecatedKeys(AdlConfKeys.java:114) at org.apache.hadoop.fs.adl.AdlFileSystem.<clinit>(AdlFileSystem.java:92) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) * Do you have any pointers for above error? Regards, Nishu -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |