Flink ML feature

[chandu soa]

Hello Community,

Can you please give me some pointers for implementing Machine Learning using Flink. 

I see Flink ML libraries were dropped in v1.9. It looks like ML feature in Flink going to be enhanced.

What is the recommended approach for implementing production grade ML based apps using Flink? v1.9 is ok?or should wait for 1.10?

Thanks,

Chandu

[Tom Blackwood]

You may try Spark ML, which is a production ready library for ML stuff.

regards.

On Tue, Dec 10, 2019 at 1:04 PM chandu soa <[hidden email]> wrote:

Hello Community,

Can you please give me some pointers for implementing Machine Learning using Flink. 

I see Flink ML libraries were dropped in v1.9. It looks like ML feature in Flink going to be enhanced.

What is the recommended approach for implementing production grade ML based apps using Flink? v1.9 is ok?or should wait for 1.10?

Thanks,

Chandu

[vino yang]

Hi Chandu,

AFAIK, there is a project named Alink[1] which is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform. FYI

Best,

Vino

[1]: https://github.com/alibaba/Alink

Tom Blackwood <[hidden email]> 于2019年12月10日周二 下午2:07写道：

You may try Spark ML, which is a production ready library for ML stuff.

regards.

On Tue, Dec 10, 2019 at 1:04 PM chandu soa <[hidden email]> wrote:

Hello Community,

Can you please give me some pointers for implementing Machine Learning using Flink. 

I see Flink ML libraries were dropped in v1.9. It looks like ML feature in Flink going to be enhanced.

What is the recommended approach for implementing production grade ML based apps using Flink? v1.9 is ok?or should wait for 1.10?

Thanks,

Chandu

[Benoît Paris-2]

Is there any information as to whether Alink is going to be contributed to Apache Flink as the official ML Lib?

On Tue, Dec 10, 2019 at 7:11 AM vino yang <[hidden email]> wrote:

Hi Chandu,

AFAIK, there is a project named Alink[1] which is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform. FYI

Best,

Vino

[1]: https://github.com/alibaba/Alink

Tom Blackwood <[hidden email]> 于2019年12月10日周二 下午2:07写道：

You may try Spark ML, which is a production ready library for ML stuff.

regards.

On Tue, Dec 10, 2019 at 1:04 PM chandu soa <[hidden email]> wrote:

Hello Community,

Can you please give me some pointers for implementing Machine Learning using Flink. 

I see Flink ML libraries were dropped in v1.9. It looks like ML feature in Flink going to be enhanced.

What is the recommended approach for implementing production grade ML based apps using Flink? v1.9 is ok?or should wait for 1.10?

Thanks,

Chandu

--

[vino yang]

Hi Benoit,

I can only try to ping [hidden email] [hidden email]  who may know more information to answer this question.

Best,

Vino

Benoît Paris <[hidden email]> 于2019年12月10日周二 下午7:06写道：

Is there any information as to whether Alink is going to be contributed to Apache Flink as the official ML Lib?

On Tue, Dec 10, 2019 at 7:11 AM vino yang <[hidden email]> wrote:

Hi Chandu,

AFAIK, there is a project named Alink[1] which is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform. FYI

Best,

Vino

[1]: https://github.com/alibaba/Alink

Tom Blackwood <[hidden email]> 于2019年12月10日周二 下午2:07写道：

You may try Spark ML, which is a production ready library for ML stuff.

regards.

On Tue, Dec 10, 2019 at 1:04 PM chandu soa <[hidden email]> wrote:

Hello Community,

Can you please give me some pointers for implementing Machine Learning using Flink. 

I see Flink ML libraries were dropped in v1.9. It looks like ML feature in Flink going to be enhanced.

What is the recommended approach for implementing production grade ML based apps using Flink? v1.9 is ok?or should wait for 1.10?

Thanks,

Chandu

--

Benoît Paris

Ingénieur Machine Learning Explicable

Tél : [phone]   

http://benoit.paris

http://explicable.ml

[Till Rohrmann]

Hi guys,

it is true that we dropped Flink-ML with 1.9. The reason is that the community started working on a new ML library which you can find under flink-ml-parent [1]. This module contains the framework for building ML pipelines but not yet too many algorithms iirc. The plan is to extend this library with algorithms from Alink in the near future to grow Flink's machine learning library.

[1] https://github.com/apache/flink/tree/master/flink-ml-parent

Cheers,

Till

On Wed, Dec 11, 2019 at 3:42 AM vino yang <[hidden email]> wrote:

Hi Benoit,

I can only try to ping [hidden email] [hidden email]  who may know more information to answer this question.

Best,

Vino

Benoît Paris <[hidden email]> 于2019年12月10日周二 下午7:06写道：

Is there any information as to whether Alink is going to be contributed to Apache Flink as the official ML Lib?

On Tue, Dec 10, 2019 at 7:11 AM vino yang <[hidden email]> wrote:

Hi Chandu,

AFAIK, there is a project named Alink[1] which is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform. FYI

Best,

Vino

[1]: https://github.com/alibaba/Alink

Tom Blackwood <[hidden email]> 于2019年12月10日周二 下午2:07写道：

You may try Spark ML, which is a production ready library for ML stuff.

regards.

On Tue, Dec 10, 2019 at 1:04 PM chandu soa <[hidden email]> wrote:

Hello Community,

Can you please give me some pointers for implementing Machine Learning using Flink. 

I see Flink ML libraries were dropped in v1.9. It looks like ML feature in Flink going to be enhanced.

What is the recommended approach for implementing production grade ML based apps using Flink? v1.9 is ok?or should wait for 1.10?

Thanks,

Chandu

--

Benoît Paris

Ingénieur Machine Learning Explicable

Tél : [phone]   

http://benoit.paris

http://explicable.ml

[Rong Rong]

Hi guys,

Yes, as Till mentioned. The community is working on a new ML library and we are working closely with the Alink project to bring the algorithms.

You can find more information regarding the new ML design architecture in FLIP-39 [1]. 

One of the major change is that the new ML library [2] will be based on the Table API [3], instead of depending on the dataset/datastream API.

I've cc-ed [hidden email], who has been a major contributor to the Alink project to provide more information.

--

Rong

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs

[2] https://github.com/apache/flink/tree/master/flink-ml-parent

[3] https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/tableApi.html

On Wed, Dec 11, 2019 at 1:11 AM Till Rohrmann <[hidden email]> wrote:

Hi guys,

it is true that we dropped Flink-ML with 1.9. The reason is that the community started working on a new ML library which you can find under flink-ml-parent [1]. This module contains the framework for building ML pipelines but not yet too many algorithms iirc. The plan is to extend this library with algorithms from Alink in the near future to grow Flink's machine learning library.

[1] https://github.com/apache/flink/tree/master/flink-ml-parent

Cheers,

Till

On Wed, Dec 11, 2019 at 3:42 AM vino yang <[hidden email]> wrote:

Hi Benoit,

I can only try to ping [hidden email] [hidden email]  who may know more information to answer this question.

Best,

Vino

Benoît Paris <[hidden email]> 于2019年12月10日周二 下午7:06写道：

Is there any information as to whether Alink is going to be contributed to Apache Flink as the official ML Lib?

On Tue, Dec 10, 2019 at 7:11 AM vino yang <[hidden email]> wrote:

Hi Chandu,

AFAIK, there is a project named Alink[1] which is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform. FYI

Best,

Vino

[1]: https://github.com/alibaba/Alink

Tom Blackwood <[hidden email]> 于2019年12月10日周二 下午2:07写道：

You may try Spark ML, which is a production ready library for ML stuff.

regards.

On Tue, Dec 10, 2019 at 1:04 PM chandu soa <[hidden email]> wrote:

Hello Community,

Can you please give me some pointers for implementing Machine Learning using Flink. 

I see Flink ML libraries were dropped in v1.9. It looks like ML feature in Flink going to be enhanced.

What is the recommended approach for implementing production grade ML based apps using Flink? v1.9 is ok?or should wait for 1.10?

Thanks,

Chandu

--

Benoît Paris

Ingénieur Machine Learning Explicable

Tél : [phone]   

http://benoit.paris

http://explicable.ml