Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Posted by shashank734 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Can-not-resolve-org-apache-hadoop-fs-Path-in-1-4-0-tp17139p17386.html

Hi Timo,

I am using Rocksdbstatebackend with hdfs path. I have following flink dependencies in my sbt :

"org.slf4j" % "slf4j-log4j12" % "1.7.21",

  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",

  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",

  "org.apache.flink" %% "flink-cep-scala" % flinkVersion,

  "org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion,

  "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion,

  "org.apache.flink" %% "flink-statebackend-rocksdb" % flinkVersion,

  "org.apache.flink" %% "flink-connector-cassandra" % "1.3.2",

  "org.apache.flink" % "flink-shaded-hadoop2" % flinkVersion,

when i start flink yarn session  it's working fine even it's creating flink checkpointing directory and copying libs into hdfs.

But when I submit the application to this yarn session it prints following logs :

Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*

But application fails contuniously with logs which i have sent earlier.

‌I have tried to add flink- hadoop-compability*.jar as suggested by Jorn but it's not working.

On Tue, Dec 19, 2017 at 5:08 PM, shashank agarwal <[hidden email]> wrote:

yes, it's working fine. now not getting compile time error.

But when i trying to run this on cluster or yarn, getting following runtime error :

org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
	at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
	at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
	at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop File System abstraction does not support scheme 'hdfs'. Either no file system implementation exists for that scheme, or the relevant classes are missing from the classpath.
	at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:102)
	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
	... 12 more
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
	at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:99)
	... 13 more

while submitting job it's printing following logs so i think it's including hdoop libs :

Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*

On Fri, Dec 8, 2017 at 9:24 PM, shashank agarwal <[hidden email]> wrote:

Sure i’ll Try that. Thanks

On Fri, 8 Dec 2017 at 9:18 PM, Stephan Ewen <[hidden email]> wrote:

I would recommend to add "flink-shaded-hadoop2". That is a bundle of all Hadoop dependencies used by Flink.

On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek <[hidden email]> wrote:

I see, thanks for letting us know!

On 8. Dec 2017, at 15:42, shashank agarwal <[hidden email]> wrote:

<img width="0" height="0" class="m_1553599686143405755m_-8226718503733193951m_5954409089651021616m_8277047368309584877mailtrack-img" alt="" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" style="display:flex">I had to include two dependencies.

hadoop-hdfs (this for HDFS configuration) 

hadoop-common (this for Path)

‌

On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek <[hidden email]> wrote:

I think hadoop-hdfs might be sufficient.

On 8. Dec 2017, at 14:48, shashank agarwal <[hidden email]> wrote:

<img width="0" height="0" class="m_1553599686143405755m_-8226718503733193951m_5954409089651021616m_8277047368309584877m_-4024719543416093672mailtrack-img" alt="" style="display:flex" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7">Can you specifically guide which dependencies should I add to extend this :

https://github.com/apache/flink/blob/release-1.4.0-rc3/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/Bucketer.java

is the Hadoop-core is sufficient. 


‌

On Fri, Dec 8, 2017 at 6:58 PM, shashank agarwal <[hidden email]> wrote:

<img width="0" height="0" class="m_1553599686143405755m_-8226718503733193951m_5954409089651021616m_8277047368309584877m_-4024719543416093672m_1305801829708615289mailtrack-img" alt="" style="display:flex" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7">It's a compilation error. I think I have to include the Hadoop dependencies.



‌

On Fri, Dec 8, 2017 at 6:54 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

Is this a compilation error or at runtime. But in general, yes you have to include the Hadoop dependencies if they're not there.

Best,

Aljoscha

On 8. Dec 2017, at 14:10, shashank agarwal <[hidden email]> wrote:

<img width="0" height="0" class="m_1553599686143405755m_-8226718503733193951m_5954409089651021616m_8277047368309584877m_-4024719543416093672m_1305801829708615289m_1570241767079666426mailtrack-img" alt="" style="display:flex" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7">Hello,

I am trying to test 1.4.0-RC3, Hadoop libraries removed in this version. Actually, i have created custom Bucketer for the bucketing sink.  I am extending 
org.apache.flink.streaming.connectors.fs.bucketing.Bucketer

in the class, i have to use org.apache.hadoop.fs.Path  but as hadoop libraries removed it's giving error 

"object hadoop is not a member of package org.apache"

Should i have to include Hadoop client libs in build.sbt dependencies.

Thanks

Shashank
‌

-- 

-- 

-- 

--

--

--