Flink Material & Papers

[Hanna Prinz]

Guten Abend everyone,

I’m currently writing a term paper about Flink at the HTW Berlin and I wanted to ask you if you can help with papers (or other material) about Flink. I could also come over to the TU if someone's doing a lecture about Flink.

And now that I’m writing you: I accidentally ran into this guide when I wanted to implement a demo for my presentation (which I know now is meant for development on the Flink Core): https://ci.apache.org/projects/flink/flink-docs-master/internals/ide_setup.html#intellij-idea

But anyway, I wanted to tell you that the Scala Compiler Plugin can’t be installed like instructed because there’s no „Install Jetbrains Plugin…“ in the Dialog (see screenshot attached).

I’m using IntelliJ IDEA 2016.2.5, Build #IU-162.2228.15, built on October 14, 2016 on macOS 10.12.1.

Many thanks in advance!

Cheers

Hanna

[Dominik Safaric]

Hi Hanna,

I would certainly recommend if you haven’t so far to check the official docs of Flink at flink.apache.org. The documentation is comprehensive and understandable. 

From that point, I would recommend the following blog posts and academic papers: 

•  Apache Flink: Stream and Batch Processing in a Single Engine - http://sites.computer.org/debull/A15dec/p28.pdf
• Lightweight Asynchronous Snapshots for Distributed Dataflows - https://arxiv.org/pdf/1506.08603v1.pdf
• https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
• https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks
• https://ci.apache.org/projects/flink/flink-docs-master/internals/job_scheduling.html

In addition to, I would suggest you to read the Realtime Data Processing at Facebook paper describing some of the important characteristics of stream processing engines generally applicable to Flink as well. 

Regards,

Dominik

On 21 Nov 2016, at 17:53, Hanna Prinz <[hidden email]> wrote:

Guten Abend everyone,

I’m currently writing a term paper about Flink at the HTW Berlin and I wanted to ask you if you can help with papers (or other material) about Flink. I could also come over to the TU if someone's doing a lecture about Flink.

And now that I’m writing you: I accidentally ran into this guide when I wanted to implement a demo for my presentation (which I know now is meant for development on the Flink Core): https://ci.apache.org/projects/flink/flink-docs-master/internals/ide_setup.html#intellij-idea

But anyway, I wanted to tell you that the Scala Compiler Plugin can’t be installed like instructed because there’s no „Install Jetbrains Plugin…“ in the Dialog (see screenshot attached).

I’m using IntelliJ IDEA 2016.2.5, Build #IU-162.2228.15, built on October 14, 2016 on macOS 10.12.1.

Many thanks in advance!

Cheers

Hanna

<Bildschirmfoto 2016-11-21 um 17.11.37.png>

[Paris Carbone]

+1 for the references

Imho the most relevant scientific publication related to the current state-of-the-art of Flink is the first one cited by Dominik (IEEE Bulletin). So it makes sense to cite that one.

However, Hanna, if you are also interested about prior work at TU Berlin that bootstrapped the Flink project check the papers on the Stratosphere system [1-4]. 

This explains and focuses on the data set (batch) processing programming model and optimisation framework that is accessible on Flink. 

[1] <a href="http://barbie.uta.edu/~hdfeng/bigdata/Papers/The Stratosphere platform for big data analytics.pdf" class="">http://barbie.uta.edu/~hdfeng/bigdata/Papers/The%20Stratosphere%20platform%20for%20big%20data%20analytics.pdf

[2] https://arxiv.org/pdf/1208.0088.pdf

[3] https://arxiv.org/pdf/1208.0087.pdf

[4] http://dl.acm.org/citation.cfm?id=2621940

Paris

On 21 Nov 2016, at 18:03, Dominik Safaric <[hidden email]> wrote:

Hi Hanna,

I would certainly recommend if you haven’t so far to check the official docs of Flink at flink.apache.org. The documentation is comprehensive and understandable. 

From that point, I would recommend the following blog posts and academic papers: 

•  Apache Flink: Stream and Batch Processing in a Single Engine - http://sites.computer.org/debull/A15dec/p28.pdf
• Lightweight Asynchronous Snapshots for Distributed Dataflows - https://arxiv.org/pdf/1506.08603v1.pdf
• https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html
• <a href="https://cwiki.apache.org/confluence/display/FLINK/Data&#43;exchange&#43;between&#43;tasks" class="">https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks
• https://ci.apache.org/projects/flink/flink-docs-master/internals/job_scheduling.html

In addition to, I would suggest you to read the Realtime Data Processing at Facebook paper describing some of the important characteristics of stream processing engines generally applicable to Flink as well. 

Regards,

Dominik

On 21 Nov 2016, at 17:53, Hanna Prinz <[hidden email]> wrote:

Guten Abend everyone,

I’m currently writing a term paper about Flink at the HTW Berlin and I wanted to ask you if you can help with papers (or other material) about Flink. I could also come over to the TU if someone's doing a lecture about Flink.

And now that I’m writing you: I accidentally ran into this guide when I wanted to implement a demo for my presentation (which I know now is meant for development on the Flink Core): https://ci.apache.org/projects/flink/flink-docs-master/internals/ide_setup.html#intellij-idea

But anyway, I wanted to tell you that the Scala Compiler Plugin can’t be installed like instructed because there’s no „Install Jetbrains Plugin…“ in the Dialog (see screenshot attached).

I’m using IntelliJ IDEA 2016.2.5, Build #IU-162.2228.15, built on October 14, 2016 on macOS 10.12.1.

Many thanks in advance!

Cheers

Hanna

<Bildschirmfoto 2016-11-21 um 17.11.37.png>