FLIP-16, FLIP-15 Status Updates?

[John Tipper]

Hi All,

Does anyone know what the current status is for FLIP-16 (loop fault tolerance) and FLIP-15 (redesign iterations) please? I can see lots of work back in 2016, but it all seemed to stop and go quiet since about March 2017. I see iterations as offering very interesting capabilities for Flink, so it would be good to understand how we can get this moving again.

Many thanks,

John

Sent from my iPhone

[Timo Walther]

Hi John,

you are right that there was not much progress in the last years around
these two FLIPs. Mostly due to shift of priorities. However, with the
big Blink code contribution from Alibaba and joint development forces
for a unified batch and streaming runtime [1], it is very likely that
also iterations and thus machine learning algorithms will see more
development efforts.

The community is working on roadmap page for the website. And I can
already reveal that a new iterations model is mentioned there. The new
Flink roadmap page can be expected in the next 2-3 weeks.

I hope this information helps.

Regards,
Timo

[1]
https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html

Am 19.02.19 um 12:47 schrieb John Tipper:
> Hi All,
>
> Does anyone know what the current status is for FLIP-16 (loop fault tolerance) and FLIP-15 (redesign iterations) please? I can see lots of work back in 2016, but it all seemed to stop and go quiet since about March 2017. I see iterations as offering very interesting capabilities for Flink, so it would be good to understand how we can get this moving again.
>
> Many thanks,
>
> John
>
> Sent from my iPhone

[John Tipper]

Hi Timo,

That’s great, thank you very much. If I’d like to contribute, is it best to wait until the roadmap has been published? And is this the best list to ask on, or is the development mailing list better?

Many thanks,

John

Sent from my iPhone

> On 19 Feb 2019, at 16:29, Timo Walther <[hidden email]> wrote:
>
> Hi John,
>
> you are right that there was not much progress in the last years around these two FLIPs. Mostly due to shift of priorities. However, with the big Blink code contribution from Alibaba and joint development forces for a unified batch and streaming runtime [1], it is very likely that also iterations and thus machine learning algorithms will see more development efforts.
>
> The community is working on roadmap page for the website. And I can already reveal that a new iterations model is mentioned there. The new Flink roadmap page can be expected in the next 2-3 weeks.
>
> I hope this information helps.
>
> Regards,
> Timo
>
> [1] https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html
>
>> Am 19.02.19 um 12:47 schrieb John Tipper:
>> Hi All,
>>
>> Does anyone know what the current status is for FLIP-16 (loop fault tolerance) and FLIP-15 (redesign iterations) please? I can see lots of work back in 2016, but it all seemed to stop and go quiet since about March 2017. I see iterations as offering very interesting capabilities for Flink, so it would be good to understand how we can get this moving again.
>>
>> Many thanks,
>>
>> John
>>
>> Sent from my iPhone
>
>

[Stephan Ewen]

Hi John!

I know some committers are working on iterations, but on a bigger update. That might subsume the FLIPs 15 and 16 eventually.

I believe they will share some part of that soon (in a few weeks).

Best,

Stephan

On Tue, Feb 19, 2019 at 5:45 PM John Tipper <[hidden email]> wrote:

Hi Timo,

That’s great, thank you very much. If I’d like to contribute, is it best to wait until the roadmap has been published? And is this the best list to ask on, or is the development mailing list better?

Many thanks,

John

Sent from my iPhone

> On 19 Feb 2019, at 16:29, Timo Walther <[hidden email]> wrote:
>
> Hi John,
>
> you are right that there was not much progress in the last years around these two FLIPs. Mostly due to shift of priorities. However, with the big Blink code contribution from Alibaba and joint development forces for a unified batch and streaming runtime [1], it is very likely that also iterations and thus machine learning algorithms will see more development efforts.
>
> The community is working on roadmap page for the website. And I can already reveal that a new iterations model is mentioned there. The new Flink roadmap page can be expected in the next 2-3 weeks.
>
> I hope this information helps.
>
> Regards,
> Timo
>
> [1] https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html
>
>> Am 19.02.19 um 12:47 schrieb John Tipper:
>> Hi All,
>>
>> Does anyone know what the current status is for FLIP-16 (loop fault tolerance) and FLIP-15 (redesign iterations) please? I can see lots of work back in 2016, but it all seemed to stop and go quiet since about March 2017. I see iterations as offering very interesting capabilities for Flink, so it would be good to understand how we can get this moving again.
>>
>> Many thanks,
>>
>> John
>>
>> Sent from my iPhone
>
>

[Paris Carbone]

I created these FLIPs a while back, sorry for being late to this discussion but I can try to elaborate. 

The idea from FLIP-16 is proven to be correct [1] (see chapter 3) and I think it is the only way to go but I have been in favour of providing channel implementations with checkpoint behaviour built in.

I do have some pretty outdated Flink forks with these patches resolved but it is not an elegant implementation. If I am not mistaken people looked into checkpointing in-transit channel state in Blink and that is a good use case to (re)use that part of the logic imho.

Regarding FLIP-15 and beyond there are a few things we have to agree before stream iterations are finalised. 

We have looked a LOT into this ([1] see chapter 6) also together with Vasia (CC) and discussed with other committers to allow for a BSP iterative model on windows and extensions to dynamic graphs, ML etc.

Some of these ideas were also planned to be integrated in Blink but I think this is still in the prototyping phase at Alibaba, please correct me if I am wrong.

I am very happy there is finally some interest in this! The FLIP discussions were dead for years. Maybe it is time to start planning it properly together, it is a super cool feature ;)

cheers

Paris

[1] http://kth.diva-portal.org/smash/get/diva2:1240814/FULLTEXT01.pdf

On 21 Feb 2019, at 19:17, Stephan Ewen <[hidden email]> wrote:

Hi John!

I know some committers are working on iterations, but on a bigger update. That might subsume the FLIPs 15 and 16 eventually.

I believe they will share some part of that soon (in a few weeks).

Best,

Stephan

On Tue, Feb 19, 2019 at 5:45 PM John Tipper <[hidden email]> wrote:

Hi Timo,

That’s great, thank you very much. If I’d like to contribute, is it best to wait until the roadmap has been published? And is this the best list to ask on, or is the development mailing list better?

Many thanks,

John

Sent from my iPhone

> On 19 Feb 2019, at 16:29, Timo Walther <[hidden email]> wrote:
>
> Hi John,
>
> you are right that there was not much progress in the last years around these two FLIPs. Mostly due to shift of priorities. However, with the big Blink code contribution from Alibaba and joint development forces for a unified batch and streaming runtime [1], it is very likely that also iterations and thus machine learning algorithms will see more development efforts.
>
> The community is working on roadmap page for the website. And I can already reveal that a new iterations model is mentioned there. The new Flink roadmap page can be expected in the next 2-3 weeks.
>
> I hope this information helps.
>
> Regards,
> Timo
>
> [1] https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html
>
>> Am 19.02.19 um 12:47 schrieb John Tipper:
>> Hi All,
>>
>> Does anyone know what the current status is for FLIP-16 (loop fault tolerance) and FLIP-15 (redesign iterations) please? I can see lots of work back in 2016, but it all seemed to stop and go quiet since about March 2017. I see iterations as offering very interesting capabilities for Flink, so it would be good to understand how we can get this moving again.
>>
>> Many thanks,
>>
>> John
>>
>> Sent from my iPhone
>
>