Machine Learning: Flink and MOA

[Christophe Salperwyck]

Hi guys,

I know there is FlinkML to do some machine learning with Flink but it works on DataSet and not on DataStream, there is also SAMOA which can run on Flink but I find it a bit too complicated.

I wanted to see if it would be easy to plug directly MOA on Flink and tried to present it in the DataKRK meetup, but I didn't have time at the end of the presentation... Nevertheless I spent a bit of time plugging Flink and MOA and I thought it might be worth sharing it in case it would be interesting for someone. I also take this opportunity to get some feedback on it from people in the Flink community if they have a bit of time to review it.

Here is the code:

https://github.com/csalperwyck/moa-flink-ozabag-example

https://github.com/csalperwyck/moa-flink-traintest-example

Many Flink methods were very convenient to plug these 2 tools :-)

Keep the good work!

Cheers,

Christophe

PS: if some people are in bigdatatechwarsaw and interested, we can discuss tomorrow :-)

[Theodore Vasiloudis]

Hello Christophe,

That's very interesting, I've been working with MOA/SAMOA recently and was considering if we could create some

easy integration with Flink.

I have a Master student this year that could do some work on this, hopefully we can create something interesting

there.

Regards,

Theodore

On Wed, Feb 21, 2018 at 7:38 PM, Christophe Salperwyck <[hidden email]> wrote:

Hi guys,

I know there is FlinkML to do some machine learning with Flink but it works on DataSet and not on DataStream, there is also SAMOA which can run on Flink but I find it a bit too complicated.

I wanted to see if it would be easy to plug directly MOA on Flink and tried to present it in the DataKRK meetup, but I didn't have time at the end of the presentation... Nevertheless I spent a bit of time plugging Flink and MOA and I thought it might be worth sharing it in case it would be interesting for someone. I also take this opportunity to get some feedback on it from people in the Flink community if they have a bit of time to review it.

Here is the code:

https://github.com/csalperwyck/moa-flink-ozabag-example

https://github.com/csalperwyck/moa-flink-traintest-example

Many Flink methods were very convenient to plug these 2 tools :-)

Keep the good work!

Cheers,

Christophe

PS: if some people are in bigdatatechwarsaw and interested, we can discuss tomorrow :-)

[Christophe Salperwyck]

Hello Theodore,

Glad to hear that there is an interest in plugging MOA with Flink!

Which part/type of classifiers of MOA would you want to plug with Flink? Let me know if you want to discuss in more details.

I guess some windowing function of MOA would be better implemented as Flink Windows (performance evaluation I would say at first). We would need to speak with Albert to see how this could be handled (change some MOA code?).

Regards,

Christophe

2018-02-24 1:03 GMT+01:00 Theodore Vasiloudis <[hidden email]>:

Hello Christophe,

That's very interesting, I've been working with MOA/SAMOA recently and was considering if we could create some

easy integration with Flink.

I have a Master student this year that could do some work on this, hopefully we can create something interesting

there.

Regards,

Theodore

On Wed, Feb 21, 2018 at 7:38 PM, Christophe Salperwyck <[hidden email]> wrote:

Hi guys,

I know there is FlinkML to do some machine learning with Flink but it works on DataSet and not on DataStream, there is also SAMOA which can run on Flink but I find it a bit too complicated.

I wanted to see if it would be easy to plug directly MOA on Flink and tried to present it in the DataKRK meetup, but I didn't have time at the end of the presentation... Nevertheless I spent a bit of time plugging Flink and MOA and I thought it might be worth sharing it in case it would be interesting for someone. I also take this opportunity to get some feedback on it from people in the Flink community if they have a bit of time to review it.

Here is the code:

https://github.com/csalperwyck/moa-flink-ozabag-example

https://github.com/csalperwyck/moa-flink-traintest-example

Many Flink methods were very convenient to plug these 2 tools :-)

Keep the good work!

Cheers,

Christophe

PS: if some people are in bigdatatechwarsaw and interested, we can discuss tomorrow :-)