(DEPRECATED) Apache Flink User Mailing List archive.

Re: flink-dist shading

Classic

List

Threaded

6 messages Options

Foster, Craig

Re: flink-dist shading

I’m not even sure this was delivered to the ‘dev’ list but I’ll go ahead and forward the same email to the ‘user’ list since I haven’t seen a response.

-------

I’m following up on the issue in FLINK-5013 about flink-dist specifically requiring Maven 3.0.5 through to <3.3. This affects people who build Flink with BigTop (not only EMR), so I’m wondering about the context and how we can properly shade the Apache HTTP libraries so that flink-dist can be built with a current version of Maven. Any insight into this would be helpful.

Thanks!

Craig

Tzu-Li (Gordon) Tai

Re: flink-dist shading

Hi Craig,

I think the email wasn't sent to the ‘dev’ list, somehow.

Have you tried this:

mvn clean install -DskipTests
# In Maven 3.3 the shading of flink-dist doesn't work properly in one run, so we need to run mvn for flink-dist again.
cd flink-dist
mvn clean install -DskipTests

I agree that it’ll affect downstream users who need to build Flink themselves, and would be best if it can be resolved.

The above is still more or less a “workaround”, but since I don’t really know the reason for why the newer Maven versions

won’t properly shade, we’ll probably need to wait for others more knowledgable on the build infrastructure to chime in and

see if there’s a good long-term solution.

Best Regards,

Gordon

On November 19, 2016 at 8:48:32 AM, Foster, Craig ([hidden email]) wrote:

I’m not even sure this was delivered to the ‘dev’ list but I’ll go ahead and forward the same email to the ‘user’ list since I haven’t seen a response.

-------

I’m following up on the issue in FLINK-5013 about flink-dist specifically requiring Maven 3.0.5 through to <3.3. This affects people who build Flink with BigTop (not only EMR), so I’m wondering about the context and how we can properly shade the Apache HTTP libraries so that flink-dist can be built with a current version of Maven. Any insight into this would be helpful.

Thanks!

Craig

rmetzger0

Re: flink-dist shading

Hi Craig,

I also received only this email (and I'm a moderator of the dev@ list, so the message never made it into Apache's infra)

When this issue was first reported [1][2] I asked on the Maven mailing list what's going on [3]. I think this JIRA contains the most information on the issue: https://issues.apache.org/jira/browse/MNG-5899

It doesn't seem that Maven is going to fix the issue anytime soon.

One idea I had regarding this issue was to print a warning in the maven output if we detect Maven 3.3 at build time.

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/flink-dist-packaging-including-unshaded-classes-td9504.html

[2] https://issues.apache.org/jira/browse/FLINK-3158

[3] http://comments.gmane.org/gmane.comp.jakarta.turbine.maven.user/137621

On Fri, Nov 18, 2016 at 7:23 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Craig,

I think the email wasn't sent to the ‘dev’ list, somehow.

Have you tried this:
mvn clean install -DskipTests
# In Maven 3.3 the shading of flink-dist doesn't work properly in one run, so we need to run mvn for flink-dist again.
cd flink-dist
mvn clean install -DskipTests
I agree that it’ll affect downstream users who need to build Flink themselves, and would be best if it can be resolved.
The above is still more or less a “workaround”, but since I don’t really know the reason for why the newer Maven versions
won’t properly shade, we’ll probably need to wait for others more knowledgable on the build infrastructure to chime in and
see if there’s a good long-term solution.

Best Regards,
Gordon
On November 19, 2016 at 8:48:32 AM, Foster, Craig ([hidden email]) wrote:

I’m not even sure this was delivered to the ‘dev’ list but I’ll go ahead and forward the same email to the ‘user’ list since I haven’t seen a response.

-------

I’m following up on the issue in FLINK-5013 about flink-dist specifically requiring Maven 3.0.5 through to <3.3. This affects people who build Flink with BigTop (not only EMR), so I’m wondering about the context and how we can properly shade the Apache HTTP libraries so that flink-dist can be built with a current version of Maven. Any insight into this would be helpful.

Thanks!

Craig

Foster, Craig

Re: flink-dist shading

Thanks for explaining, Robert and Gordon. For however it helps, I’ll comment on the original Maven issue which seems to be affecting many people. I am trying to create RPMs using BigTop and am still not quite sure how to handle this case. I guess what I could do is build the parent first in our script and then build flink-dist again as described in the instructions. I’ll try that out and see if it resolves the issue.

From: Robert Metzger <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Saturday, November 19, 2016 at 4:08 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: flink-dist shading

Hi Craig,

I also received only this email (and I'm a moderator of the dev@ list, so the message never made it into Apache's infra)

It doesn't seem that Maven is going to fix the issue anytime soon.

One idea I had regarding this issue was to print a warning in the maven output if we detect Maven 3.3 at build time.

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/flink-dist-packaging-including-unshaded-classes-td9504.html

[2] https://issues.apache.org/jira/browse/FLINK-3158

[3] http://comments.gmane.org/gmane.comp.jakarta.turbine.maven.user/137621

On Fri, Nov 18, 2016 at 7:23 PM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Craig,

I think the email wasn't sent to the ‘dev’ list, somehow.

Have you tried this:
mvn clean install -DskipTests
# In Maven 3.3 the shading of flink-dist doesn't work properly in one run, so we need to run mvn for flink-dist again.
cd flink-dist
mvn clean install -DskipTests
I agree that it’ll affect downstream users who need to build Flink themselves, and would be best if it can be resolved.

The above is still more or less a “workaround”, but since I don’t really know the reason for why the newer Maven versions

won’t properly shade, we’ll probably need to wait for others more knowledgable on the build infrastructure to chime in and

see if there’s a good long-term solution.

Best Regards,

Gordon

On November 19, 2016 at 8:48:32 AM, Foster, Craig ([hidden email]) wrote:

I’m not even sure this was delivered to the ‘dev’ list but I’ll go ahead and forward the same email to the ‘user’ list since I haven’t seen a response.

-------

I’m following up on the issue in FLINK-5013 about flink-dist specifically requiring Maven 3.0.5 through to <3.3. This affects people who build Flink with BigTop (not only EMR), so I’m wondering about the context and how we can properly shade the Apache HTTP libraries so that flink-dist can be built with a current version of Maven. Any insight into this would be helpful.

Thanks!

Craig

Maximilian Michels

Re: flink-dist shading

Hi Craig,

I've left a comment on the original Maven JIRA issue to revive the
discussion. For BigTop, you can handle this in the build script by
building flink-dist again after a successful build. That will always
work independently of the Maven 3.x version.

-Max

On Mon, Nov 21, 2016 at 6:27 PM, Foster, Craig <[hidden email]> wrote:

> Thanks for explaining, Robert and Gordon. For however it helps, I’ll comment
> on the original Maven issue which seems to be affecting many people. I am
> trying to create RPMs using BigTop and am still not quite sure how to handle
> this case. I guess what I could do is build the parent first in our script
> and then build flink-dist again as described in the instructions. I’ll try
> that out and see if it resolves the issue.
>
>
>
> From: Robert Metzger <[hidden email]>
> Reply-To: "[hidden email]" <[hidden email]>
> Date: Saturday, November 19, 2016 at 4:08 AM
> To: "[hidden email]" <[hidden email]>
> Subject: Re: flink-dist shading
>
>
>
> Hi Craig,
>
> I also received only this email (and I'm a moderator of the dev@ list, so
> the message never made it into Apache's infra)
>
>
>
> When this issue was first reported [1][2] I asked on the Maven mailing list
> what's going on [3]. I think this JIRA contains the most information on the
> issue: https://issues.apache.org/jira/browse/MNG-5899
>
> It doesn't seem that Maven is going to fix the issue anytime soon.
>
>
>
> One idea I had regarding this issue was to print a warning in the maven
> output if we detect Maven 3.3 at build time.
>
>
>
>
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/flink-dist-packaging-including-unshaded-classes-td9504.html
>
> [2] https://issues.apache.org/jira/browse/FLINK-3158
>
> [3] http://comments.gmane.org/gmane.comp.jakarta.turbine.maven.user/137621
>
>
>
>
>
>
>
> On Fri, Nov 18, 2016 at 7:23 PM, Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> Hi Craig,
>
>
>
> I think the email wasn't sent to the ‘dev’ list, somehow.
>
>
>
> Have you tried this:
>
>
>
> mvn clean install -DskipTests
>
> # In Maven 3.3 the shading of flink-dist doesn't work properly in one run,
> so we need to run mvn for flink-dist again.
>
> cd flink-dist
>
> mvn clean install -DskipTests
>
> I agree that it’ll affect downstream users who need to build Flink
> themselves, and would be best if it can be resolved.
>
> The above is still more or less a “workaround”, but since I don’t really
> know the reason for why the newer Maven versions
>
> won’t properly shade, we’ll probably need to wait for others more
> knowledgable on the build infrastructure to chime in and
>
> see if there’s a good long-term solution.
>
>
>
> Best Regards,
>
> Gordon
>
> On November 19, 2016 at 8:48:32 AM, Foster, Craig ([hidden email])
> wrote:
>
> I’m not even sure this was delivered to the ‘dev’ list but I’ll go ahead and
> forward the same email to the ‘user’ list since I haven’t seen a response.
>
> -------
>
>
>
> I’m following up on the issue in FLINK-5013 about flink-dist specifically
> requiring Maven 3.0.5 through to <3.3. This affects people who build Flink
> with BigTop (not only EMR), so I’m wondering about the context and how we
> can properly shade the Apache HTTP libraries so that flink-dist can be built
> with a current version of Maven. Any insight into this would be helpful.
>
>
>
> Thanks!
>
> Craig
>
>
>
>

Foster, Craig

Re: flink-dist shading

Thanks Max. Yep, I just confirmed it works.

On 11/22/16, 2:09 AM, "Maximilian Michels" <[hidden email]> wrote:

Hi Craig,

I've left a comment on the original Maven JIRA issue to revive the
discussion. For BigTop, you can handle this in the build script by
building flink-dist again after a successful build. That will always
work independently of the Maven 3.x version.

-Max

On Mon, Nov 21, 2016 at 6:27 PM, Foster, Craig <[hidden email]> wrote:
> Thanks for explaining, Robert and Gordon. For however it helps, I’ll comment
> on the original Maven issue which seems to be affecting many people. I am
> trying to create RPMs using BigTop and am still not quite sure how to handle
> this case. I guess what I could do is build the parent first in our script
> and then build flink-dist again as described in the instructions. I’ll try
> that out and see if it resolves the issue.
>
>
>
> From: Robert Metzger <[hidden email]>
> Reply-To: "[hidden email]" <[hidden email]>
> Date: Saturday, November 19, 2016 at 4:08 AM
> To: "[hidden email]" <[hidden email]>
> Subject: Re: flink-dist shading
>
>
>
> Hi Craig,
>
> I also received only this email (and I'm a moderator of the dev@ list, so
> the message never made it into Apache's infra)
>
>
>
> When this issue was first reported [1][2] I asked on the Maven mailing list
> what's going on [3]. I think this JIRA contains the most information on the
> issue: https://issues.apache.org/jira/browse/MNG-5899
>
> It doesn't seem that Maven is going to fix the issue anytime soon.
>
>
>
> One idea I had regarding this issue was to print a warning in the maven
> output if we detect Maven 3.3 at build time.
>
>
>
>
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/flink-dist-packaging-including-unshaded-classes-td9504.html
>
> [2] https://issues.apache.org/jira/browse/FLINK-3158
>
> [3] http://comments.gmane.org/gmane.comp.jakarta.turbine.maven.user/137621
>
>
>
>
>
>
>
> On Fri, Nov 18, 2016 at 7:23 PM, Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> Hi Craig,
>
>
>
> I think the email wasn't sent to the ‘dev’ list, somehow.
>
>
>
> Have you tried this:
>
>
>
> mvn clean install -DskipTests
>
> # In Maven 3.3 the shading of flink-dist doesn't work properly in one run,
> so we need to run mvn for flink-dist again.
>
> cd flink-dist
>
> mvn clean install -DskipTests
>
> I agree that it’ll affect downstream users who need to build Flink
> themselves, and would be best if it can be resolved.
>
> The above is still more or less a “workaround”, but since I don’t really
> know the reason for why the newer Maven versions
>
> won’t properly shade, we’ll probably need to wait for others more
> knowledgable on the build infrastructure to chime in and
>
> see if there’s a good long-term solution.
>
>
>
> Best Regards,
>
> Gordon
>
> On November 19, 2016 at 8:48:32 AM, Foster, Craig ([hidden email])
> wrote:
>
> I’m not even sure this was delivered to the ‘dev’ list but I’ll go ahead and
> forward the same email to the ‘user’ list since I haven’t seen a response.
>
> -------
>
>
>
> I’m following up on the issue in FLINK-5013 about flink-dist specifically
> requiring Maven 3.0.5 through to <3.3. This affects people who build Flink
> with BigTop (not only EMR), so I’m wondering about the context and how we
> can properly shade the Apache HTTP libraries so that flink-dist can be built
> with a current version of Maven. Any insight into this would be helpful.
>
>
>
> Thanks!
>
> Craig
>
>
>
>