Does Flink release Hadoop(R) 2.8 work with Hadoop 3?

[Mich Talebzadeh]

Hi,

I can run Flink without bundled Hadoop fine. I was wondering if Flink with Hadoop 2.8 works with Hadoop 3 as well?

Thanks,

[vino yang]

Hi Mich,

I think this depends on the backward compatibility of the Hadoop client API. In theory, there is no problem. 

Hadoop 2.8 to Hadoop 3.0 is a very large upgrade, and personally recommend using a client version that is consistent with the Hadoop cluster. 

By compiling and packaging from source, you can let Flink bundle specific Hadoop versions. Flink has shaded hadoop see here[1]. You can try to change the maven version and repackage the project.

[1]: https://github.com/apache/flink/tree/master/flink-shaded-hadoop

Thanks, vino.

2018-07-27 23:31 GMT+08:00 Mich Talebzadeh <[hidden email]>:

Hi,

I can run Flink without bundled Hadoop fine. I was wondering if Flink with Hadoop 2.8 works with Hadoop 3 as well?

Thanks,

[Mich Talebzadeh]

Hi Vino,

Many thanks.

I can confirm that Flink version flink-1.5.0-bin-hadoop28-scala_2.11 works fine with Hadoop 3.1.0. I did not need to build it from source.

Regards,

On Sat, 28 Jul 2018 at 07:48, vino yang <[hidden email]> wrote:

Hi Mich,

I think this depends on the backward compatibility of the Hadoop client API. In theory, there is no problem. 

Hadoop 2.8 to Hadoop 3.0 is a very large upgrade, and personally recommend using a client version that is consistent with the Hadoop cluster. 

By compiling and packaging from source, you can let Flink bundle specific Hadoop versions. Flink has shaded hadoop see here[1]. You can try to change the maven version and repackage the project.

[1]: https://github.com/apache/flink/tree/master/flink-shaded-hadoop

Thanks, vino.

2018-07-27 23:31 GMT+08:00 Mich Talebzadeh <[hidden email]>:

Hi,

I can run Flink without bundled Hadoop fine. I was wondering if Flink with Hadoop 2.8 works with Hadoop 3 as well?

Thanks,