Exception while submitting jobs through Yarn

[Till Rohrmann]

Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?

On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <[hidden email]> wrote:

I don't think I can access core-default as it comes with Hadoop jar

On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <[hidden email]> wrote:

Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it.

On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <[hidden email]> wrote:

Hi,

After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1

log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());

Also, following is the response of $ locate xerces command on the server -

/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar

Now, I can say that the version of xerces are same.

So, what is causing this issue if Xerces version is in sync?

I am very excited to discover the issue :)

Thanks,

On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <[hidden email]> wrote:

Could you check which xerces version you have on your classpath? Apparently, it cannot read core-default.xml as Ted pointed out. This might be the root cause for the failure.

Cheers,

Till

On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <[hidden email]> wrote:

Hi,

Sorry for the confusion, but the yarn is running on Hadoop version 2.7 only and hence I am using Flink 1.5 Hadoop 2.7 binary.

Below are the details provided by Yarn version command :

Hadoop 2.7.3.2.6.3.0-235

Subversion [hidden email]:hortonworks/hadoop.git -r 45bfd33bba8acadfa0e6024c80981c023b28d454

Compiled by jenkins on 2017-10-30T02:31Z

Compiled with protoc 2.5.0

From source with checksum cd1a4a466ef450f547c279989f3aa3

This command was run using /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar

Please let me know if you have found the resolution to my issue :)

Thanks,

On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <[hidden email]> wrote:

Which Hadoop version have you installed? It looks as if Flink has been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink binaries or the one built for Hadoop 2.6.

Cheers,

Till

On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <[hidden email]> wrote:

Ok, I have attached the log file.

Please check and let me know.

Thanks,

On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <[hidden email]> wrote:

Hi Gravit,

I think Till is interested to know about classpath details present at the start of JM and TM logs e.g. following logs provide classpath details used by TM in our case.

2018-06-17 19:01:30,656 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  - --------------------------------------------------------------------------------
2018-06-17 19:01:30,658 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Starting YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC)
2018-06-17 19:01:30,659 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  OS current user: yarn
2018-06-17 19:01:31,662 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Current Hadoop/Kerberos user: hadoop
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Maximum heap size: 6647 MiBytes
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-06-17 19:01:31,664 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Hadoop version: 2.8.3
2018-06-17 19:01:31,664 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JVM Options:
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Xms6936m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Xmx6936m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -XX:MaxDirectMemorySize=4072m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlogback.configurationFile=file:./logback.xml
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlog4j.configuration=file:./log4j.properties
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Program Arguments:
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     --configDir
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     .

2018-06-17 19:01:31,666 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........

--

Thanks,

Amit

On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <[hidden email]> wrote:

Hi,

Please refer to my previous mail for complete logs.

Thanks,

On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <[hidden email]> wrote:

Could you also please share the complete log file with us.

Cheers,

Till

On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <[hidden email]> wrote:

The error for core-default.xml is interesting. 

Flink doesn't have this file. Probably it came with Yarn. Please check the hadoop version Flink was built with versus the hadoop version in your cluster.

Thanks

-------- Original message --------

From: Garvit Sharma <[hidden email]>

Date: 6/16/18 7:23 AM (GMT-08:00)

To: [hidden email]

Cc: Chesnay Schepler <[hidden email]>, [hidden email]

Subject: Re: Exception while submitting jobs through Yarn

I am not able to figure out, got stuck badly in this since last 1 week. Any little help would be appreciated.

2018-06-16 19:25:10,523 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 8

2018-06-16 19:25:10,578 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 1

2018-06-16 19:25:10,588 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8

2018-06-16 19:25:10,591 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 5

2018-06-16 19:25:10,597 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8

2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration                          - error parsing conf core-default.xml

javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.

at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)

at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)

at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)

at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)

at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)

at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)

at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)

at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)

at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)

at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)

2018-06-16 19:25:10,620 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Error while getting queue information from YARN: null

2018-06-16 19:25:10,621 DEBUG org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Error details

java.lang.ExceptionInInitializerError

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)

at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)

at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)

--

--

--

--

--

[Garvit Sharma]

Yes, it is. 

On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <[hidden email]> wrote:

Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?

On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <[hidden email]> wrote:

I don't think I can access core-default as it comes with Hadoop jar

On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <[hidden email]> wrote:

Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it.

On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <[hidden email]> wrote:

Hi,

After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1

log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());

Also, following is the response of $ locate xerces command on the server -

/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar

Now, I can say that the version of xerces are same.

So, what is causing this issue if Xerces version is in sync?

I am very excited to discover the issue :)

Thanks,

On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <[hidden email]> wrote:

Could you check which xerces version you have on your classpath? Apparently, it cannot read core-default.xml as Ted pointed out. This might be the root cause for the failure.

Cheers,

Till

On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <[hidden email]> wrote:

Hi,

Sorry for the confusion, but the yarn is running on Hadoop version 2.7 only and hence I am using Flink 1.5 Hadoop 2.7 binary.

Below are the details provided by Yarn version command :

Hadoop 2.7.3.2.6.3.0-235

Subversion [hidden email]:hortonworks/hadoop.git -r 45bfd33bba8acadfa0e6024c80981c023b28d454

Compiled by jenkins on 2017-10-30T02:31Z

Compiled with protoc 2.5.0

From source with checksum cd1a4a466ef450f547c279989f3aa3

This command was run using /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar

Please let me know if you have found the resolution to my issue :)

Thanks,

On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <[hidden email]> wrote:

Which Hadoop version have you installed? It looks as if Flink has been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink binaries or the one built for Hadoop 2.6.

Cheers,

Till

On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <[hidden email]> wrote:

Ok, I have attached the log file.

Please check and let me know.

Thanks,

On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <[hidden email]> wrote:

Hi Gravit,

I think Till is interested to know about classpath details present at the start of JM and TM logs e.g. following logs provide classpath details used by TM in our case.

2018-06-17 19:01:30,656 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  - --------------------------------------------------------------------------------
2018-06-17 19:01:30,658 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Starting YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC)
2018-06-17 19:01:30,659 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  OS current user: yarn
2018-06-17 19:01:31,662 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Current Hadoop/Kerberos user: hadoop
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Maximum heap size: 6647 MiBytes
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-06-17 19:01:31,664 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Hadoop version: 2.8.3
2018-06-17 19:01:31,664 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JVM Options:
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Xms6936m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Xmx6936m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -XX:MaxDirectMemorySize=4072m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlogback.configurationFile=file:./logback.xml
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlog4j.configuration=file:./log4j.properties
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Program Arguments:
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     --configDir
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     .

2018-06-17 19:01:31,666 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........

--

Thanks,

Amit

On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <[hidden email]> wrote:

Hi,

Please refer to my previous mail for complete logs.

Thanks,

On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <[hidden email]> wrote:

Could you also please share the complete log file with us.

Cheers,

Till

On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <[hidden email]> wrote:

The error for core-default.xml is interesting. 

Flink doesn't have this file. Probably it came with Yarn. Please check the hadoop version Flink was built with versus the hadoop version in your cluster.

Thanks

-------- Original message --------

From: Garvit Sharma <[hidden email]>

Date: 6/16/18 7:23 AM (GMT-08:00)

To: [hidden email]

Cc: Chesnay Schepler <[hidden email]>, [hidden email]

Subject: Re: Exception while submitting jobs through Yarn

I am not able to figure out, got stuck badly in this since last 1 week. Any little help would be appreciated.

2018-06-16 19:25:10,523 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 8

2018-06-16 19:25:10,578 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 1

2018-06-16 19:25:10,588 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8

2018-06-16 19:25:10,591 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 5

2018-06-16 19:25:10,597 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8

2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration                          - error parsing conf core-default.xml

javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.

at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)

at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)

at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)

at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)

at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)

at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)

at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)

at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)

at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)

at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)

2018-06-16 19:25:10,620 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Error while getting queue information from YARN: null

2018-06-16 19:25:10,621 DEBUG org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Error details

java.lang.ExceptionInInitializerError

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)

at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)

at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)

--

--

--

--

--

--

[Garvit Sharma]

Any help on this?

On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <[hidden email]> wrote:

Yes, it is. 

On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <[hidden email]> wrote:

Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?

On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <[hidden email]> wrote:

I don't think I can access core-default as it comes with Hadoop jar

On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <[hidden email]> wrote:

Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it.

On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <[hidden email]> wrote:

Hi,

After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1

log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());

Also, following is the response of $ locate xerces command on the server -

/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar

Now, I can say that the version of xerces are same.

So, what is causing this issue if Xerces version is in sync?

I am very excited to discover the issue :)

Thanks,

On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <[hidden email]> wrote:

Could you check which xerces version you have on your classpath? Apparently, it cannot read core-default.xml as Ted pointed out. This might be the root cause for the failure.

Cheers,

Till

On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <[hidden email]> wrote:

Hi,

Sorry for the confusion, but the yarn is running on Hadoop version 2.7 only and hence I am using Flink 1.5 Hadoop 2.7 binary.

Below are the details provided by Yarn version command :

Hadoop 2.7.3.2.6.3.0-235

Subversion [hidden email]:hortonworks/hadoop.git -r 45bfd33bba8acadfa0e6024c80981c023b28d454

Compiled by jenkins on 2017-10-30T02:31Z

Compiled with protoc 2.5.0

From source with checksum cd1a4a466ef450f547c279989f3aa3

This command was run using /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar

Please let me know if you have found the resolution to my issue :)

Thanks,

On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <[hidden email]> wrote:

Which Hadoop version have you installed? It looks as if Flink has been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink binaries or the one built for Hadoop 2.6.

Cheers,

Till

On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <[hidden email]> wrote:

Ok, I have attached the log file.

Please check and let me know.

Thanks,

On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <[hidden email]> wrote:

Hi Gravit,

I think Till is interested to know about classpath details present at the start of JM and TM logs e.g. following logs provide classpath details used by TM in our case.

2018-06-17 19:01:30,656 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  - --------------------------------------------------------------------------------
2018-06-17 19:01:30,658 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Starting YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC)
2018-06-17 19:01:30,659 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  OS current user: yarn
2018-06-17 19:01:31,662 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Current Hadoop/Kerberos user: hadoop
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Maximum heap size: 6647 MiBytes
2018-06-17 19:01:31,663 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-06-17 19:01:31,664 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Hadoop version: 2.8.3
2018-06-17 19:01:31,664 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  JVM Options:
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Xms6936m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Xmx6936m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -XX:MaxDirectMemorySize=4072m
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlogback.configurationFile=file:./logback.xml
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     -Dlog4j.configuration=file:./log4j.properties
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Program Arguments:
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     --configDir
2018-06-17 19:01:31,665 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -     .

2018-06-17 19:01:31,666 INFO  org.apache.flink.yarn.YarnTaskExecutorRunner                  -  Classpath: lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........

--

Thanks,

Amit

On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <[hidden email]> wrote:

Hi,

Please refer to my previous mail for complete logs.

Thanks,

On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <[hidden email]> wrote:

Could you also please share the complete log file with us.

Cheers,

Till

On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <[hidden email]> wrote:

The error for core-default.xml is interesting. 

Flink doesn't have this file. Probably it came with Yarn. Please check the hadoop version Flink was built with versus the hadoop version in your cluster.

Thanks

-------- Original message --------

From: Garvit Sharma <[hidden email]>

Date: 6/16/18 7:23 AM (GMT-08:00)

To: [hidden email]

Cc: Chesnay Schepler <[hidden email]>, [hidden email]

Subject: Re: Exception while submitting jobs through Yarn

I am not able to figure out, got stuck badly in this since last 1 week. Any little help would be appreciated.

2018-06-16 19:25:10,523 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 8

2018-06-16 19:25:10,578 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 1

2018-06-16 19:25:10,588 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8

2018-06-16 19:25:10,591 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - Parallelism set: 1 for 5

2018-06-16 19:25:10,597 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator  - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8

2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration                          - error parsing conf core-default.xml

javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.

at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)

at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)

at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)

at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)

at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)

at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)

at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)

at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)

at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)

at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)

2018-06-16 19:25:10,620 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Error while getting queue information from YARN: null

2018-06-16 19:25:10,621 DEBUG org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Error details

java.lang.ExceptionInInitializerError

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)

at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)

at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)

at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)

at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)

at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)

at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)

--

--

--

--

--

--

--

[Ted Yu]

Since you're using a vendor's distro, I would suggest asking on their user forum.

Cheers

-------- Original message --------

From: Garvit Sharma <[hidden email]>

Date: 6/19/18 3:34 AM (GMT-08:00)

To: [hidden email]

Cc: Amit Jain <[hidden email]>, Chesnay Schepler <[hidden email]>, Ted Yu <[hidden email]>, [hidden email]

Subject: Re: Exception while submitting jobs through Yarn

Any help on this?

On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <[hidden email]> wrote:

Yes, it is. 

On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <[hidden email]> wrote:

Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?

On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <[hidden email]> wrote:

I don't think I can access core-default as it comes with Hadoop jar

On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <[hidden email]> wrote:

Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it.

On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <[hidden email]> wrote:

Hi,

After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1

log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());

Also, following is the response of $ locate xerces command on the server -

/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar

[Garvit Sharma]

So, finally, I have got this working. The issue was because of a poor library which was using xerces 2.6 :).

In this process, I found few things missing from the doc would like to contribute the same.

I really appreciate the support provided.

Thanks,

On Tue, 19 Jun 2018 at 4:05 PM, Ted Yu <[hidden email]> wrote:

Since you're using a vendor's distro, I would suggest asking on their user forum.

Cheers

-------- Original message --------

From: Garvit Sharma <[hidden email]>

Date: 6/19/18 3:34 AM (GMT-08:00)

To: [hidden email]

Cc: Amit Jain <[hidden email]>, Chesnay Schepler <[hidden email]>, Ted Yu <[hidden email]>, [hidden email]

Subject: Re: Exception while submitting jobs through Yarn

Any help on this?

On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <[hidden email]> wrote:

Yes, it is. 

On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <[hidden email]> wrote:

Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?

On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <[hidden email]> wrote:

I don't think I can access core-default as it comes with Hadoop jar

On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <[hidden email]> wrote:

Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it.

On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <[hidden email]> wrote:

Hi,

After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1

log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());

Also, following is the response of $ locate xerces command on the server -

/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar

[Till Rohrmann]

Great to hear. Please open a PR for the improvements you like to contribute.

Cheers,

Till

On Wed, Jun 20, 2018 at 4:56 PM Garvit Sharma <[hidden email]> wrote:

So, finally, I have got this working. The issue was because of a poor library which was using xerces 2.6 :).

In this process, I found few things missing from the doc would like to contribute the same.

I really appreciate the support provided.

Thanks,

On Tue, 19 Jun 2018 at 4:05 PM, Ted Yu <[hidden email]> wrote:

Since you're using a vendor's distro, I would suggest asking on their user forum.

Cheers

-------- Original message --------

From: Garvit Sharma <[hidden email]>

Date: 6/19/18 3:34 AM (GMT-08:00)

To: [hidden email]

Cc: Amit Jain <[hidden email]>, Chesnay Schepler <[hidden email]>, Ted Yu <[hidden email]>, [hidden email]

Subject: Re: Exception while submitting jobs through Yarn

Any help on this?

On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <[hidden email]> wrote:

Yes, it is. 

On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <[hidden email]> wrote:

Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?

On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <[hidden email]> wrote:

I don't think I can access core-default as it comes with Hadoop jar

On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <[hidden email]> wrote:

Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it.

On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <[hidden email]> wrote:

Hi,

After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1

log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());

Also, following is the response of $ locate xerces command on the server -

/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar

/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar

/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar

/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar

/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar

/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar