Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Posted by Stephan Ewen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Can-not-resolve-org-apache-hadoop-fs-Path-in-1-4-0-tp17139p17154.html

I would recommend to add "flink-shaded-hadoop2". That is a bundle of all Hadoop dependencies used by Flink.

On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek <[hidden email]> wrote:

I see, thanks for letting us know!

On 8. Dec 2017, at 15:42, shashank agarwal <[hidden email]> wrote:

I had to include two dependencies.

hadoop-hdfs (this for HDFS configuration) 

hadoop-common (this for Path)

‌

On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek <[hidden email]> wrote:

I think hadoop-hdfs might be sufficient.

On 8. Dec 2017, at 14:48, shashank agarwal <[hidden email]> wrote:

Can you specifically guide which dependencies should I add to extend this :

https://github.com/apache/flink/blob/release-1.4.0-rc3/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/Bucketer.java

is the Hadoop-core is sufficient. 


‌

On Fri, Dec 8, 2017 at 6:58 PM, shashank agarwal <[hidden email]> wrote:

It's a compilation error. I think I have to include the Hadoop dependencies.



‌

On Fri, Dec 8, 2017 at 6:54 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

Is this a compilation error or at runtime. But in general, yes you have to include the Hadoop dependencies if they're not there.

Best,

Aljoscha

On 8. Dec 2017, at 14:10, shashank agarwal <[hidden email]> wrote:

Hello,

I am trying to test 1.4.0-RC3, Hadoop libraries removed in this version. Actually, i have created custom Bucketer for the bucketing sink.  I am extending 
org.apache.flink.streaming.connectors.fs.bucketing.Bucketer

in the class, i have to use org.apache.hadoop.fs.Path  but as hadoop libraries removed it's giving error 

"object hadoop is not a member of package org.apache"

Should i have to include Hadoop client libs in build.sbt dependencies.

Thanks

Shashank
‌

-- 

-- 

--