(DEPRECATED) Apache Flink User Mailing List archive.

Return of Flink shading problems in 1.2.0

classic Classic list List threaded Threaded
♦
♦
 7 messages Options
Foster, Craig
Reply | Threaded
Open this post in threaded view
♦
♦
|

Return of Flink shading problems in 1.2.0

Foster, Craig

Hi:

A few months ago, I was building Flink and ran into shading issues for flink-dist as described in your docs. We resolved this in BigTop by adding the correct way to build flink-dist in the do-component-build script and everything was fine after that.

 

Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m trying to figure out what’s changed and how to fix it. Here’s how the flink-dist jar looks with proper shading:

 

jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep HttpConnectionParams
2485 Tue Jan 01 00:00:00 UTC 1980 org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
3479 Tue Jan 01 00:00:00 UTC 1980 org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class

 

When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the RPM:

 

jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
2392 Tue Jan 01 00:00:00 GMT 1980 org/apache/commons/httpclient/params/HttpConnectionParams.class
2485 Tue Jan 01 00:00:00 GMT 1980 org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
3479 Tue Jan 01 00:00:00 GMT 1980 org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
2868 Tue Jan 01 00:00:00 GMT 1980 org/apache/http/params/HttpConnectionParams.class

 

I thought maybe it was some strange thing going on with BigTop, so then I tried just straight building Flink 1.2.0 (outside BigTop) and get the same shading:

 

jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams

  2485 Fri Mar 17 05:41:16 GMT 2017 org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class

  3479 Fri Mar 17 05:41:16 GMT 2017 org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class

  2392 Fri Mar 17 05:41:24 GMT 2017 org/apache/commons/httpclient/params/HttpConnectionParams.class

  2868 Fri Mar 17 05:41:24 GMT 2017 org/apache/http/params/HttpConnectionParams.class

 

And, yes, this is after going into flink-dist and running mvn clean install again since I am using Maven 3.3.x.

 

Here’s a snippet from my Maven version:

mvn -version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)

Maven home: /usr/local/apache-maven

Java version: 1.8.0_121, vendor: Oracle Corporation

 

Any ideas on what my problem might be here?

 

Thanks,

Craig

 
Ufuk Celebi
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Return of Flink shading problems in 1.2.0

Ufuk Celebi
Pulling in Robert and Stephan who know the project's shading setup the best.

On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <[hidden email]> wrote:

> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
>   2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>   2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Return of Flink shading problems in 1.2.0

Stephan Ewen
Hi Craig!

Maven 3.3.x has a shading problem. You need to build two times, once from root, once inside "flink-dist". Have a look here:


Maybe that way missed in BigTop?

I am wondering if we should actually throw an error if building with Maven 3.3.x - too many users run into that issue.

Stephan



On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <[hidden email]> wrote:
Pulling in Robert and Stephan who know the project's shading setup the best.

On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <[hidden email]> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
>   2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>   2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>
Foster, Craig
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Return of Flink shading problems in 1.2.0

Foster, Craig
Hey Stephen: 
I am building twice in every case described in my previous mail. Well, building then rebuilding the flink-dist submodule.

This was fixed in BigTop but I started seeing this issue again with Flink 1.2.0. I was wondering if there's something else in the environment that could prevent the shading from working because it isn't now even with the workaround.

On Mar 17, 2017, at 4:08 AM, Stephan Ewen <[hidden email]> wrote:

Hi Craig!

Maven 3.3.x has a shading problem. You need to build two times, once from root, once inside "flink-dist". Have a look here:


Maybe that way missed in BigTop?

I am wondering if we should actually throw an error if building with Maven 3.3.x - too many users run into that issue.

Stephan



On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <[hidden email]> wrote:
Pulling in Robert and Stephan who know the project's shading setup the best.

On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <[hidden email]> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
>   2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>   2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>
Foster, Craig
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Return of Flink shading problems in 1.2.0

Foster, Craig

Ping. So I’ve built with 3.0.5 and it does give proper shading. So it does get me yet another workaround where my only recourse is to use a max version of Maven. Still, I feel there should be a long-term fix at some point in time.

 

I also believe there is a regression in Flink 1.2.0 for Maven 3.3.x with the process as documented, so hoping someone can at least duplicate or let me know of a new workaround for 3.3.x.

 

Thanks!

Craig

 

From: "Foster, Craig" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, March 17, 2017 at 7:23 AM
To: "[hidden email]" <[hidden email]>
Cc: Ufuk Celebi <[hidden email]>, Robert Metzger <[hidden email]>, Stephan Ewen <[hidden email]>
Subject: Re: Return of Flink shading problems in 1.2.0

 

Hey Stephen: 

I am building twice in every case described in my previous mail. Well, building then rebuilding the flink-dist submodule.

 

This was fixed in BigTop but I started seeing this issue again with Flink 1.2.0. I was wondering if there's something else in the environment that could prevent the shading from working because it isn't now even with the workaround.


On Mar 17, 2017, at 4:08 AM, Stephan Ewen <[hidden email]> wrote:

Hi Craig!

 

Maven 3.3.x has a shading problem. You need to build two times, once from root, once inside "flink-dist". Have a look here:

 

 

Maybe that way missed in BigTop?

 

I am wondering if we should actually throw an error if building with Maven 3.3.x - too many users run into that issue.

 

Stephan

 

 

 

On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <[hidden email]> wrote:

Pulling in Robert and Stephan who know the project's shading setup the best.


On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <[hidden email]> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
>   2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>   2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>

 
rmetzger0
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Return of Flink shading problems in 1.2.0

rmetzger0
Hi Craig,

I was able to reproduce the issue with maven 3.3 in Flink 1.2. I'll look into it.

On Fri, Mar 17, 2017 at 11:56 PM, Foster, Craig <[hidden email]> wrote:

Ping. So I’ve built with 3.0.5 and it does give proper shading. So it does get me yet another workaround where my only recourse is to use a max version of Maven. Still, I feel there should be a long-term fix at some point in time.

 

I also believe there is a regression in Flink 1.2.0 for Maven 3.3.x with the process as documented, so hoping someone can at least duplicate or let me know of a new workaround for 3.3.x.

 

Thanks!

Craig

 

From: "Foster, Craig" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, March 17, 2017 at 7:23 AM
To: "[hidden email]" <[hidden email]>
Cc: Ufuk Celebi <[hidden email]>, Robert Metzger <[hidden email]>, Stephan Ewen <[hidden email]>
Subject: Re: Return of Flink shading problems in 1.2.0

 

Hey Stephen: 

I am building twice in every case described in my previous mail. Well, building then rebuilding the flink-dist submodule.

 

This was fixed in BigTop but I started seeing this issue again with Flink 1.2.0. I was wondering if there's something else in the environment that could prevent the shading from working because it isn't now even with the workaround.


On Mar 17, 2017, at 4:08 AM, Stephan Ewen <[hidden email]> wrote:

Hi Craig!

 

Maven 3.3.x has a shading problem. You need to build two times, once from root, once inside "flink-dist". Have a look here:

 

 

Maybe that way missed in BigTop?

 

I am wondering if we should actually throw an error if building with Maven 3.3.x - too many users run into that issue.

 

Stephan

 

 

 

On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <[hidden email]> wrote:

Pulling in Robert and Stephan who know the project's shading setup the best.


On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <[hidden email]> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
>   2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>   2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>

 
rmetzger0
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Return of Flink shading problems in 1.2.0

rmetzger0

On Mon, Mar 20, 2017 at 10:27 AM, Robert Metzger <[hidden email]> wrote:
Hi Craig,

I was able to reproduce the issue with maven 3.3 in Flink 1.2. I'll look into it.

On Fri, Mar 17, 2017 at 11:56 PM, Foster, Craig <[hidden email]> wrote:

Ping. So I’ve built with 3.0.5 and it does give proper shading. So it does get me yet another workaround where my only recourse is to use a max version of Maven. Still, I feel there should be a long-term fix at some point in time.

 

I also believe there is a regression in Flink 1.2.0 for Maven 3.3.x with the process as documented, so hoping someone can at least duplicate or let me know of a new workaround for 3.3.x.

 

Thanks!

Craig

 

From: "Foster, Craig" <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, March 17, 2017 at 7:23 AM
To: "[hidden email]" <[hidden email]>
Cc: Ufuk Celebi <[hidden email]>, Robert Metzger <[hidden email]>, Stephan Ewen <[hidden email]>
Subject: Re: Return of Flink shading problems in 1.2.0

 

Hey Stephen: 

I am building twice in every case described in my previous mail. Well, building then rebuilding the flink-dist submodule.

 

This was fixed in BigTop but I started seeing this issue again with Flink 1.2.0. I was wondering if there's something else in the environment that could prevent the shading from working because it isn't now even with the workaround.


On Mar 17, 2017, at 4:08 AM, Stephan Ewen <[hidden email]> wrote:

Hi Craig!

 

Maven 3.3.x has a shading problem. You need to build two times, once from root, once inside "flink-dist". Have a look here:

 

 

Maybe that way missed in BigTop?

 

I am wondering if we should actually throw an error if building with Maven 3.3.x - too many users run into that issue.

 

Stephan

 

 

 

On Fri, Mar 17, 2017 at 8:14 AM, Ufuk Celebi <[hidden email]> wrote:

Pulling in Robert and Stephan who know the project's shading setup the best.


On Fri, Mar 17, 2017 at 6:52 AM, Foster, Craig <[hidden email]> wrote:
> Hi:
>
> A few months ago, I was building Flink and ran into shading issues for
> flink-dist as described in your docs. We resolved this in BigTop by adding
> the correct way to build flink-dist in the do-component-build script and
> everything was fine after that.
>
>
>
> Now, I’m running into issues doing the same now in Flink 1.2.0 and I’m
> trying to figure out what’s changed and how to fix it. Here’s how the
> flink-dist jar looks with proper shading:
>
>
>
> jar -tvf /usr/lib/flink/lib/flink-dist_2.10-1.1.4.jar | grep
> HttpConnectionParams
> 2485 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 UTC 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>
>
> When I build Flink 1.2.0 in BigTop, here’s shading for the jar found in the
> RPM:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
> 2392 Tue Jan 01 00:00:00 GMT 1980
> org/apache/commons/httpclient/params/HttpConnectionParams.class
> 2485 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
> 3479 Tue Jan 01 00:00:00 GMT 1980
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
> 2868 Tue Jan 01 00:00:00 GMT 1980
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> I thought maybe it was some strange thing going on with BigTop, so then I
> tried just straight building Flink 1.2.0 (outside BigTop) and get the same
> shading:
>
>
>
> jar -tvf flink-dist_2.10-1.2.0.jar | grep HttpConnectionParams
>
>   2485 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   3479 Fri Mar 17 05:41:16 GMT 2017
> org/apache/flink/hadoop/shaded/org/apache/http/params/HttpConnectionParams.class
>
>   2392 Fri Mar 17 05:41:24 GMT 2017
> org/apache/commons/httpclient/params/HttpConnectionParams.class
>
>   2868 Fri Mar 17 05:41:24 GMT 2017
> org/apache/http/params/HttpConnectionParams.class
>
>
>
> And, yes, this is after going into flink-dist and running mvn clean install
> again since I am using Maven 3.3.x.
>
>
>
> Here’s a snippet from my Maven version:
>
> mvn -version
>
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T16:41:47+00:00)
>
> Maven home: /usr/local/apache-maven
>
> Java version: 1.8.0_121, vendor: Oracle Corporation
>
>
>
> Any ideas on what my problem might be here?
>
>
>
> Thanks,
>
> Craig
>
>

 
Free forum by Nabble Edit this page