Re: flink-dist shading

Posted by Foster, Craig on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Re-flink-dist-shading-tp10221p10255.html

Thanks for explaining, Robert and Gordon. For however it helps, I’ll comment on the original Maven issue which seems to be affecting many people. I am trying to create RPMs using BigTop and am still not quite sure how to handle this case. I guess what I could do is build the parent first in our script and then build flink-dist again as described in the instructions. I’ll try that out and see if it resolves the issue.  

 

From: Robert Metzger <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Saturday, November 19, 2016 at 4:08 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: flink-dist shading

 

Hi Craig,

I also received only this email (and I'm a moderator of the dev@ list, so the message never made it into Apache's infra)

 

When this issue was first reported [1][2] I asked on the Maven mailing list what's going on [3]. I think this JIRA contains the most information on the issue: https://issues.apache.org/jira/browse/MNG-5899

It doesn't seem that Maven is going to fix the issue anytime soon.

 

One idea I had regarding this issue was to print a warning in the maven output if we detect Maven 3.3 at build time.

 

 

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/flink-dist-packaging-including-unshaded-classes-td9504.html

[2] https://issues.apache.org/jira/browse/FLINK-3158 

[3] http://comments.gmane.org/gmane.comp.jakarta.turbine.maven.user/137621 

 

 

 

On Fri, Nov 18, 2016 at 7:23 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Craig,

 

I think the email wasn't sent to the ‘dev’ list, somehow.

 

Have you tried this:

 

mvn clean install -DskipTests

# In Maven 3.3 the shading of flink-dist doesn't work properly in one run, so we need to run mvn for flink-dist again.

cd flink-dist

mvn clean install -DskipTests

I agree that it’ll affect downstream users who need to build Flink themselves, and would be best if it can be resolved.

The above is still more or less a “workaround”, but since I don’t really know the reason for why the newer Maven versions

won’t properly shade, we’ll probably need to wait for others more knowledgable on the build infrastructure to chime in and

see if there’s a good long-term solution.

 

Best Regards,

Gordon

On November 19, 2016 at 8:48:32 AM, Foster, Craig ([hidden email]) wrote:

I’m not even sure this was delivered to the ‘dev’ list but I’ll go ahead and forward the same email to the ‘user’ list since I haven’t seen a response.

-------

 

I’m following up on the issue in FLINK-5013 about flink-dist specifically requiring Maven 3.0.5 through to <3.3. This affects people who build Flink with BigTop (not only EMR), so I’m wondering about the context and how we can properly shade the Apache HTTP libraries so that flink-dist can be built with a current version of Maven. Any insight into this would be helpful.

 

Thanks!

Craig