Questions on GSoC project: Query optimisation layer for Flink Streaming

[bwepngong]

CONTENTS DELETED

The author has deleted this message.

[Gyula Fóra-2]

Hey,

Of course the aim of the project would not be to implement all possible optimizations because that would be impossible to do so in such short time :)

It would be nice if one could carefully select some optimizations that would make the most impact on the performance and implement those.

Regards,

Gyula

On Fri, Mar 20, 2015 at 5:15 AM, Wepngong Benaiah <[hidden email]> wrote:

hello ,
I have been making some research on
https://issues.apache.org/jira/browse/FLINK-1617 using
http://hirzels.com/martin/papers/csur14-streamopt.pdf and others. I
find out that there are many optimization techniques available like
1. OPERATOR REORDERING
2. REDUNDANCY ELIMINATION
3. OPERATOR SEPARATION
4. FUSION
5. FISSION
6. LOAD BALANCING
7. STATE SHARING

My question is do I need to choose 1 or 2 of the algorithms and
implement for GSOC or im required to implement all the algorithms
given the tight contraint for GSOC timeline
Need help @Gyula Fora

--
Wepngong Ngeh Benaiah

"Black holes are where God divided by zero. "

[Paris Carbone]

Hi Wepngong,

This is an interesting proposal. There are indeed many streaming optimisations out there but as Gyula said we should focus on a few and engineer them in a nice way. Perhaps for the time being it makes sense to focus on a streaming job graph optimiser that basically applies optimisations by statically analysing the graph before submitting it ie. query re-writing, operator reordering, operator sharing, intermediate result sharing etc. A runtime optimiser that can do things like load balancing and online reconfiguration would certainly be the next step.

cheers

Paris

On 20 Mar 2015, at 12:48, Gyula Fóra <[hidden email]> wrote:

Hey,

Of course the aim of the project would not be to implement all possible optimizations because that would be impossible to do so in such short time :)

It would be nice if one could carefully select some optimizations that would make the most impact on the performance and implement those.

Regards,

Gyula

On Fri, Mar 20, 2015 at 5:15 AM, Wepngong Benaiah <[hidden email]> wrote:

hello ,
I have been making some research on
https://issues.apache.org/jira/browse/FLINK-1617 using
http://hirzels.com/martin/papers/csur14-streamopt.pdf and others. I
find out that there are many optimization techniques available like
1. OPERATOR REORDERING
2. REDUNDANCY ELIMINATION
3. OPERATOR SEPARATION
4. FUSION
5. FISSION
6. LOAD BALANCING
7. STATE SHARING

My question is do I need to choose 1 or 2 of the algorithms and
implement for GSOC or im required to implement all the algorithms
given the tight contraint for GSOC timeline
Need help @Gyula Fora

--
Wepngong Ngeh Benaiah

"Black holes are where God divided by zero. "

[bwepngong]

CONTENTS DELETED

The author has deleted this message.