Error while setting up flink on HDInsight cluster

[nishutayal]

Hi,

I am trying to run flink on top on HDInsight cluster.  So far I have added all libraries in classpath and set YARN_CONF_DIR, HADOOP_CONF_DIR and HADOOP_CLASSPATH. It's running on hadoop 2.7.3.

When I run yarn-session.sh, It throws following error.

2017-11-21 13:07:01,946 INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl     - Timeline service address: http://headnodehost:8188/ws/v1/timeline/

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V

        at org.apache.hadoop.fs.adl.AdlConfKeys.addDeprecatedKeys(AdlConfKeys.java:114)

        at org.apache.hadoop.fs.adl.AdlFileSystem.<clinit>(AdlFileSystem.java:92)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:348)

        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)

        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)

        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)

        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)

        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)

        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)

        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)

        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)

        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)

        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.startAppMaster(AbstractYarnClusterDescriptor.java:609)

        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:582)

        at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:441)

        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)

        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)

        at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)

        at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)

        at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)

I decompiled the hadoop jar to see, if reloadExistingConfiguration method is present in Configuration class. That's available too. 

Has anyone faced the same before? Any help is really appreciated. Thanks in advance.

--