Roadmap for Execution Mode (Batch/Streaming) and interaction with Table/SQL APIs

[burkaygur]

Hi Flink Community,

Really excited for the "true unification" of Batch and Stream APIs, and had
a few questions for clarification after reading the release notes, and
watching the video by aljoscha  here
<https://www.youtube.com/watch?v=z9ye4jzp4DQ>  .


1) How do these changes impact the Table and SQL APIs? Are they completely
orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
as well?
2) What is the best ticket to follow the roadmap & track the progress of
this whole project. Specifically the parts about bootstrapping of state. I
would love to help contribute to it.

Best,
Burkay



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Piotr Nowojski-4]

Hi,

1. I think those changes will mostly bring new features/functionalities to the existing Streaming APIs in order to fully support batch executions. For example one way or another to better handle "bounded data streams" in the DataStream API.

2. I think there is and there is not going to be one single umbrella ticket, as this whole idea is going to take quite a bit of time to fully design. There are a couple of the initial design proposals connected with the batch and streaming unification:

FLIP-134 [1]

FLIP-147 [2]

But I'm pretty sure more will follow in the future.

Best,

Piotrek

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API

[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-147%3A+Support+Checkpoints+After+Tasks+Finished

śr., 6 sty 2021 o 16:20 burkaygur <[hidden email]> napisał(a):

Hi Flink Community,

Really excited for the "true unification" of Batch and Stream APIs, and had
a few questions for clarification after reading the release notes, and
watching the video by aljoscha  here
<https://www.youtube.com/watch?v=z9ye4jzp4DQ>  .


1) How do these changes impact the Table and SQL APIs? Are they completely
orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
as well?
2) What is the best ticket to follow the roadmap & track the progress of
this whole project. Specifically the parts about bootstrapping of state. I
would love to help contribute to it.

Best,
Burkay



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Aljoscha Krettek]

Also cc'ing [hidden email]

On 2021/01/06 09:19, burkaygur wrote:
>1) How do these changes impact the Table and SQL APIs? Are they completely
>orthogonal or can we get the benefits of the new Batch Mode with Flink SQL
>as well?

The answer here is a bit complicated. The Table API/SQL already use
similar techniques under the hood for BATCH execution. Our recent
changes just made similar functionality available for the DataStream
API. So in a way the changes are orthogonal.

However, the changes are relevant when you want to interoperate with the
DataStream and Table API. There it becomes relevant that both parts can
do STREAMING/BATCH execution well. We're not 100% there yet on this
front but we're tracking some work under FLIP-136 [1].

[1] https://cwiki.apache.org/confluence/x/0DN4CQ

>2) What is the best ticket to follow the roadmap & track the progress of
>this whole project. Specifically the parts about bootstrapping of state. I
>would love to help contribute to it.

I would say the best way to follow progress is the dev mailing list and
the the FLIP overview page [2]. That's not super intuitive and can be
hard to follow for outsiders. Sometimes, people such as myself will
write blog posts on the Flink website or private blogs that try and
shine a light on the development so it might help to follow some
relevant people from the project on Twitter, where such posts are often
announced.

Specifically about state bootstrapping, we don't have many concrete
thoughts yet. It would help if you could talk about some of the
requirements that you would have for this.

Best,
Aljoscha

[2]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Best,
Aljoscha