FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

[Stephan Ewen]

Hi all!

We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism.

I think it is a pretty interesting read for people that are interested in Flink or data streaming in general.

The blog post talks about:

  - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots.

  - Performance of Flink, throughput, latency, and tradeoffs.

  - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers.

Comments welcome!

Greetings,

Stephan

[Stephan Ewen]

I forgot the link ;-)

http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/

On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:

Hi all!

We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism.

I think it is a pretty interesting read for people that are interested in Flink or data streaming in general.

The blog post talks about:

  - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots.

  - Performance of Flink, throughput, latency, and tradeoffs.

  - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers.

Comments welcome!

Greetings,

Stephan

[hawin]

Great job, Guys

Let me read it carefully. 

On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <[hidden email]> wrote:

I forgot the link ;-)

http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/

On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:

Hi all!

We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism.

I think it is a pretty interesting read for people that are interested in Flink or data streaming in general.

The blog post talks about:

  - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots.

  - Performance of Flink, throughput, latency, and tradeoffs.

  - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers.

Comments welcome!

Greetings,

Stephan

[Ankur Chauhan]

Pretty awesome piece. 

Sent from my iPhone

On Aug 5, 2015, at 10:10, Hawin Jiang <[hidden email]> wrote:

Great job, Guys

Let me read it carefully. 

On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <[hidden email]> wrote:

I forgot the link ;-)

http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/

On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:

Hi all!

We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism.

I think it is a pretty interesting read for people that are interested in Flink or data streaming in general.

The blog post talks about:

  - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots.

  - Performance of Flink, throughput, latency, and tradeoffs.

  - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers.

Comments welcome!

Greetings,

Stephan