(DEPRECATED) Apache Flink User Mailing List archive.

Build Flink against a vendor specific Hadoop version

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
Elise RAMÉ
Reply | Threaded
Open this post in threaded view
♦
♦
|

Build Flink against a vendor specific Hadoop version

Elise RAMÉ
Hi all,

I need to build Flink 1.9 against a vendor specific Hadoop version and something bother me in the Building Flink from Source documentation (1.9 and 1.10-SNAPSHOT versions).

As far as I understand this documentation, I have to build flink-shaded using the vendor-repos Maven profile : 
mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.5.0-292

I tried it using flink-shaded 7.0 (given version in Flink downloads page), 8.0 version or master branch and always got the same error (attached screenshot) :

It seems to me that flink-shaded-hadoop module has been moved from Flink to flink-shaded recently and I found the expected profile in Flink pom.xml file :
So I succeed in building flink-shaded and then Flink against my specific version of Hadoop by adding this profile into flink-shaded pom.xml first.

Didn’t I understand the documentation or vendor-repos profile has indeed to be defined into flink-shaded pom.xml ?

Thanks, 
Elise
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Build Flink against a vendor specific Hadoop version

Stephan Ewen
The easiest thing is to build Flink against a specific Hadoop version at all, but just to take plain Flink (Hadoop free) and export the HADOOP_CLASSPATH variable to point to the vendor libraries.

Does that work for you?

On Thu, Aug 29, 2019 at 4:15 PM Elise RAMÉ <[hidden email]> wrote:
Hi all,

I need to build Flink 1.9 against a vendor specific Hadoop version and something bother me in the Building Flink from Source documentation (1.9 and 1.10-SNAPSHOT versions).

As far as I understand this documentation, I have to build flink-shaded using the vendor-repos Maven profile : 
mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.5.0-292

I tried it using flink-shaded 7.0 (given version in Flink downloads page), 8.0 version or master branch and always got the same error (attached screenshot) :

It seems to me that flink-shaded-hadoop module has been moved from Flink to flink-shaded recently and I found the expected profile in Flink pom.xml file :
So I succeed in building flink-shaded and then Flink against my specific version of Hadoop by adding this profile into flink-shaded pom.xml first.

Didn’t I understand the documentation or vendor-repos profile has indeed to be defined into flink-shaded pom.xml ?

Thanks, 
Elise
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Build Flink against a vendor specific Hadoop version

Chesnay Schepler
It appears we did not port the vendor-repos profile to flink-shaded.

If the classpath option is not viable in your case (for example because you cannot use child-first classloading), then currently you would have to setup the vendor repos yourself in your local maven installation.

On 29/08/2019 18:48, Stephan Ewen wrote:
The easiest thing is to build Flink against a specific Hadoop version at all, but just to take plain Flink (Hadoop free) and export the HADOOP_CLASSPATH variable to point to the vendor libraries.

Does that work for you?

On Thu, Aug 29, 2019 at 4:15 PM Elise RAMÉ <[hidden email]> wrote:
Hi all,

I need to build Flink 1.9 against a vendor specific Hadoop version and something bother me in the Building Flink from Source documentation (1.9 and 1.10-SNAPSHOT versions).

As far as I understand this documentation, I have to build flink-shaded using the vendor-repos Maven profile : 
mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.5.0-292

I tried it using flink-shaded 7.0 (given version in Flink downloads page), 8.0 version or master branch and always got the same error (attached screenshot) :

It seems to me that flink-shaded-hadoop module has been moved from Flink to flink-shaded recently and I found the expected profile in Flink pom.xml file :
So I succeed in building flink-shaded and then Flink against my specific version of Hadoop by adding this profile into flink-shaded pom.xml first.

Didn’t I understand the documentation or vendor-repos profile has indeed to be defined into flink-shaded pom.xml ?

Thanks, 
Elise
Elise RAMÉ
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Build Flink against a vendor specific Hadoop version

Elise RAMÉ
Thank you all !
Classpath option works for me and is easier so I’ll do this way.

About flink-shaded and vendor-repos, would it be helpful if I describe this issue in a Jira ticket ?

Le 30 août 2019 à 11:43, Chesnay Schepler <[hidden email]> a écrit :

It appears we did not port the vendor-repos profile to flink-shaded.

If the classpath option is not viable in your case (for example because you cannot use child-first classloading), then currently you would have to setup the vendor repos yourself in your local maven installation.

On 29/08/2019 18:48, Stephan Ewen wrote:
The easiest thing is to build Flink against a specific Hadoop version at all, but just to take plain Flink (Hadoop free) and export the HADOOP_CLASSPATH variable to point to the vendor libraries.

Does that work for you?

On Thu, Aug 29, 2019 at 4:15 PM Elise RAMÉ <[hidden email]> wrote:
Hi all,

I need to build Flink 1.9 against a vendor specific Hadoop version and something bother me in the Building Flink from Source documentation (1.9 and 1.10-SNAPSHOT versions).

As far as I understand this documentation, I have to build flink-shaded using the vendor-repos Maven profile : 
mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.5.0-292

I tried it using flink-shaded 7.0 (given version in Flink downloads page), 8.0 version or master branch and always got the same error (attached screenshot) :
<maven-error.png>

It seems to me that flink-shaded-hadoop module has been moved from Flink to flink-shaded recently and I found the expected profile in Flink pom.xml file :
So I succeed in building flink-shaded and then Flink against my specific version of Hadoop by adding this profile into flink-shaded pom.xml first.

Didn’t I understand the documentation or vendor-repos profile has indeed to be defined into flink-shaded pom.xml ?

Thanks, 
Elise
Chesnay Schepler
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Build Flink against a vendor specific Hadoop version

Chesnay Schepler
Yes, please open a JIRA ticket.

On 30/08/2019 18:46, Elise RAMÉ wrote:
Thank you all !
Classpath option works for me and is easier so I’ll do this way.

About flink-shaded and vendor-repos, would it be helpful if I describe this issue in a Jira ticket ?

Le 30 août 2019 à 11:43, Chesnay Schepler <[hidden email]> a écrit :

It appears we did not port the vendor-repos profile to flink-shaded.

If the classpath option is not viable in your case (for example because you cannot use child-first classloading), then currently you would have to setup the vendor repos yourself in your local maven installation.

On 29/08/2019 18:48, Stephan Ewen wrote:
The easiest thing is to build Flink against a specific Hadoop version at all, but just to take plain Flink (Hadoop free) and export the HADOOP_CLASSPATH variable to point to the vendor libraries.

Does that work for you?

On Thu, Aug 29, 2019 at 4:15 PM Elise RAMÉ <[hidden email]> wrote:
Hi all,

I need to build Flink 1.9 against a vendor specific Hadoop version and something bother me in the Building Flink from Source documentation (1.9 and 1.10-SNAPSHOT versions).

As far as I understand this documentation, I have to build flink-shaded using the vendor-repos Maven profile : 
mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.5.0-292

I tried it using flink-shaded 7.0 (given version in Flink downloads page), 8.0 version or master branch and always got the same error (attached screenshot) :
<maven-error.png>

It seems to me that flink-shaded-hadoop module has been moved from Flink to flink-shaded recently and I found the expected profile in Flink pom.xml file :
So I succeed in building flink-shaded and then Flink against my specific version of Hadoop by adding this profile into flink-shaded pom.xml first.

Didn’t I understand the documentation or vendor-repos profile has indeed to be defined into flink-shaded pom.xml ?

Thanks, 
Elise
Free forum by Nabble Edit this page