how Flink Optimizer work and what is process do it?

[hagersaleh]

how Flink Optimizer work and what is process do it?

[Alexander Alexandrov]

The flink optimizer will basically do two things for you (+ some other magic regarding iterations which I will leave aside for now)

1) Algorithm selection: e.g., choose between a  hash-join or merge-join;

2) Partitioning strategy: track grouping key, and if partitioning is needed, pick a suitable strategy (e.g. re-partition vs broadcast for a join);

Besides that the optimizer will fix the hyperparameters for your algorithms (e.g. memory allocation) and will ultimately generate the final program that has to be executed.

[1] http://en.wikipedia.org/wiki/Hash_join
[2] http://en.wikipedia.org/wiki/Sort-merge_join

2015-05-25 20:27 GMT+02:00 hagersaleh <[hidden email]>:

how Flink Optimizer work and what is process do it?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/how-Flink-Optimizer-work-and-what-is-process-do-it-tp1359.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[hagersaleh]

very thanks
what meaning Optimizer in flink?

[Fabian Hueske-2]

It's similar to a query optimizer in a relational database system.

2015-05-26 10:10 GMT+02:00 hagersaleh <[hidden email]>:

very thanks
what meaning Optimizer in flink?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/how-Flink-Optimizer-work-and-what-is-process-do-it-tp1359p1361.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[hagersaleh]

this optimizer automatic mode or I determination it

[Fabian Hueske-2]

Optimization happens automatically when you submit a batch program (DataSet API).

2015-05-26 10:27 GMT+02:00 hagersaleh <[hidden email]>:

this optimizer automatic mode or I determination it



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/how-Flink-Optimizer-work-and-what-is-process-do-it-tp1359p1363.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.