Loading... |
Reply to author |
Edit post |
Move post |
Delete this post |
Delete this post and replies |
Change post date |
Print post |
Permalink |
Raw mail |
databricks.com/blog/2017/06/06/simple-super-fast-streaming-engine-apache-spark.html
Should flink users be worry about this huge difference? End of microbatch with 65M benchmark. |
Loading... |
Reply to author |
Edit post |
Move post |
Delete this post |
Delete this post and replies |
Change post date |
Print post |
Permalink |
Raw mail |
From my understanding, the benchmark was done using Structured Streaming
that is still based on micro batching. There are not throughput numbers for the new "Continuous Processing" model Spark want to introduce. Only some latency numbers. Also note, that the new "Continuous Processing" will not give exactly-once semantics but only at-least-once (at least initially). Thus, there is some tradeoff to make using "Continuous Processing" once it's available. -Matthias On 06/18/2017 03:51 PM, nragon wrote: > databricks.com/blog/2017/06/06/simple-super-fast-streaming-engine-apache-spark.html > > Should flink users be worry about this huge difference? > End of microbatch with 65M benchmark. > > > > -- > View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Latest-spark-yahoo-benchmark-tp13820.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com. > |
Loading... |
Reply to author |
Edit post |
Move post |
Delete this post |
Delete this post and replies |
Change post date |
Print post |
Permalink |
Raw mail |
In reply to this post by nragon
@nragon - I think this is a classical "benchmarketing" post. A few thoughts on that
- Everyone can tune their system to be best. We ran Flink with even higher throughput than that: https://image.slidesharecdn.com/benchmark-mapr-160407212254/95/extending-the-yahoo-streaming-benchmark-mapr-benchmarks-36-638.jpg?cb=1460064274 - The benchmark is testing a complete toy case, with a few KBs of state. It says little about most interesting use cases with larger state. - The Flink system runs the benchmark with a low latency engine that is also capable of exactly once guarantees. A thing that does not exist the other compared systems. ==> Test your personal application in Flink and Spark, with respect to performance, robustness to load spikes and backpressure, ability to operate/migrate/update, etc. I am pretty sure you will find that Flink has a lot of very powerful unique aspects to it for streaming applications. On Mon, Jun 19, 2017 at 12:51 AM, nragon <[hidden email]> wrote: databricks.com/blog/2017/06/ |
Loading... |
Reply to author |
Edit post |
Move post |
Delete this post |
Delete this post and replies |
Change post date |
Print post |
Permalink |
Raw mail |
Yes, indeed.
That's why we choose Flink instead all the others. This post was just pure curiosity to see spark trying to migrate into a pure streaming engine. |
Free forum by Nabble | Edit this page |