Re: Lazy unrolling of iterative algorithms

Posted by Asterios Katsifodimos on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Lazy-unrolling-of-iterative-algorithms-tp244p246.html

We woudl like to have asap actually.

Our deadline is on Thursday. However, I think I could potentially work on the JobGraph.

What we would like to see is the effect of re-deploying a job (a la Spark) on every iteration and measure it in large deployments (i.e. the overhead of not persisting tasks on machines).

Do you think we could "simulate" this in an easy way?

Best,

Asterios

On Thursday, October 30, 2014, Stephan Ewen <[hidden email]> wrote:

Hey!

The Java API does not yet support it, the JobGraph supports it, though.

In the runtime, it will still execute everything from the start if you "attach" another rolled out version of the loop."

We should have that working in a few weeks.

What is your schedule? How fast do you need it?

Stephan

On Thu, Oct 30, 2014 at 11:43 AM, Asterios Katsifodimos <[hidden email]> wrote:

Hello,

I would like to use the feature of lazy unrolling in order to run KMeans.  

Would this be possible in he current API? Or should I use a lower level API (what we used to call Nephele?)

Thanks,

Asterios