What / Where / When / How questions in Spark 2.0 ?

[Ovidiu-Cristian MARCU]

Hi,

We can see in [2] many interesting (and expected!) improvements (promises) like extended SQL support, unified API (DataFrames, DataSets), improved engine (Tungsten relates to ideas from modern compilers and MPP databases - similar to Flink [3]), structured streaming etc. It seems we somehow assist at a smart unification of Big Data analytics (Spark, Flink - best of two worlds)!

How does Spark respond to the missing What/Where/When/How questions (capabilities) highlighted in the unified model Beam [1] ?

Best,

Ovidiu

[1] https://cloud.google.com/blog/big-data/2016/05/why-apache-beam-a-google-perspective

[2] https://databricks.com/blog/2016/05/11/spark-2-0-technical-preview-easier-faster-and-smarter.html

[3] http://stratosphere.eu/project/publications/