Re: [DISCUSS] Towards a leaner flink-dist
Posted by
Becket Qin on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/DISCUSS-Towards-a-leaner-flink-dist-tp25615p25794.html
Hi Chesnay,
Thanks for the proposal. +1 for make the distribution thinner.
Meanwhile, it would be useful to have all the peripheral libraries/jars hosted somewhere so users can download them from a centralized place. We can also encourage the community to contribute their libraries, such as connectors and other pluggables, to the same place (maybe a separate category), so the community can share the commonly used libraries as well.
Thanks,
Jiangjie (Becket) Qin
Hi Chesnay,
Thanks a lot for the proposal! +1 for a leaner flink-dist and improve the
"Download" page.
I think a leaner flink-dist would be very helpful. If we bundle all jars
into a single one, this will easily cause class conflict problem.
Best,
Hequn
On Fri, Jan 25, 2019 at 2:48 PM jincheng sun <[hidden email]>
wrote:
> Hi Chesnay,
>
> Thank you for the proposal. And i like it very much.
>
> +1 for the leaner distribution.
>
> About improve the "Download" page, I think we can add the connectors
> download link in the "Optional components" section which @Timo Walther
> <[hidden email]> mentioned above.
>
>
> Regards,
> Jincheng
>
> Chesnay Schepler <[hidden email]> 于2019年1月18日周五 下午5:59写道：
>
>> Hello,
>>
>> the binary distribution that we release by now contains quite a lot of
>> optional components, including various filesystems, metric reporters and
>> libraries. Most users will only use a fraction of these, and as such
>> pretty much only increase the size of flink-dist.
>>
>> With Flink growing more and more in scope I don't believe it to be
>> feasible to ship everything we have with every distribution, and instead
>> suggest more of a "pick-what-you-need" model, where flink-dist is rather
>> lean and additional components are downloaded separately and added by
>> the user.
>>
>> This would primarily affect the /opt directory, but could also be
>> extended to cover flink-dist. For example, the yarn and mesos code could
>> be spliced out into separate jars that could be added to lib manually.
>>
>> Let me know what you think.
>>
>> Regards,
>>
>> Chesnay
>>
>>