MXNet on Flink

[Guy Needham]

MXNet (incubating), https://mxnet.incubator.apache.org/, is a pretty cool deep learning framework. It allows the user to develop across languages and to share the model model objects between languages. This is useful as it allows us to develop models in Python, then export the model to a file which for example a Spark job can pick up and continue to train the model in a distributed setting. This flexibility and additionally the high performance of the library differentiate it from other frameworks.

In Spark, the driver process contains a parameter server allowing worker processes to communicate information on the model training asynchronously which is a common pattern in the distribution of model training. Source can be found here: https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark

I think it would be useful to be able to use MXNet with Flink in the same way one can use MXNet with Spark to train a model in a distributed setting on a large stream of data.

Is anyone else interested in this?

[flinkuser101]

This ia dope.

On 11/19/17 at 5:10 PM, Guy wrote:

MXNet (incubating),  https://mxnet.incubator.apache.org/, is a pretty cool deep learning framework. It allows the user to develop across languages and to share the model model objects between languages. This is useful as it allows us to develop models in Python, then export the model to a file which for example a Spark job can pick up and continue to train the model in a distributed setting. This flexibility and additionally the high performance of the library differentiate it from other frameworks.

In Spark, the driver process contains a parameter server allowing worker processes to communicate information on the model training asynchronously which is a common pattern in the distribution of model training. Source can be found here:  https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark

I think it would be useful to be able to use MXNet with Flink in the same way one can use MXNet with Spark to train a model in a distributed setting on a large stream of data.

Is anyone else interested in this?