Size of state for any known production use case

[RKandoji]

Hi Team,

I've done a POC using Flink and planning to give a presentation about my learnings and share the benefits of using Flink.

I understand that companies are using Flink to handle Tera Bytes of state, but it would be great if you could point me to any reference of a company using Flink production for a known amount of state. Or any other related links where I can get these details?

Basically I want to provide the known maximum limit of state that can be stored. This is needed because my use case requires performing stream joins on unbounded data (although data is unbounded, its not going to be super huge like 10TB)

Thanks,

Reva

[Marta Paes Moreira]

Hi, Reva.

If you are looking for the maximum known state size, I believe Alibaba is using Flink at the largest scale in production [1].

There are also other examples of variable scale scattered across Flink Forward talks [2]. In particular, this Netflix talk [3] should be interesting to you.

Marta

[1] https://www.itnextsummit.com/wp-content/uploads/2019/11/Stephan_Ewen_Stream_Processing_Beyond_Streaming.pdf (Slide 3)
[2] https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA/videos
[3] https://www.youtube.com/watch?v=2C44mUPlx5o

On Wed, Feb 12, 2020 at 10:42 PM RKandoji <[hidden email]> wrote:

Hi Team,

I've done a POC using Flink and planning to give a presentation about my learnings and share the benefits of using Flink.

I understand that companies are using Flink to handle Tera Bytes of state, but it would be great if you could point me to any reference of a company using Flink production for a known amount of state. Or any other related links where I can get these details?

Basically I want to provide the known maximum limit of state that can be stored. This is needed because my use case requires performing stream joins on unbounded data (although data is unbounded, its not going to be super huge like 10TB)

Thanks,

Reva

[RKandoji]

hey Marta,

Thank you!

This is going to be pretty useful, let me go over these and get back to you if I have any questions.

RK

On Thu, Feb 13, 2020 at 6:21 AM Marta Paes Moreira <[hidden email]> wrote:

Hi, Reva.

If you are looking for the maximum known state size, I believe Alibaba is using Flink at the largest scale in production [1].

There are also other examples of variable scale scattered across Flink Forward talks [2]. In particular, this Netflix talk [3] should be interesting to you.

Marta

[1] https://www.itnextsummit.com/wp-content/uploads/2019/11/Stephan_Ewen_Stream_Processing_Beyond_Streaming.pdf (Slide 3)
[2] https://www.youtube.com/channel/UCY8_lgiZLZErZPF47a2hXMA/videos
[3] https://www.youtube.com/watch?v=2C44mUPlx5o

On Wed, Feb 12, 2020 at 10:42 PM RKandoji <[hidden email]> wrote:

Hi Team,

I've done a POC using Flink and planning to give a presentation about my learnings and share the benefits of using Flink.

I understand that companies are using Flink to handle Tera Bytes of state, but it would be great if you could point me to any reference of a company using Flink production for a known amount of state. Or any other related links where I can get these details?

Basically I want to provide the known maximum limit of state that can be stored. This is needed because my use case requires performing stream joins on unbounded data (although data is unbounded, its not going to be super huge like 10TB)

Thanks,

Reva