How many events can Flink process each second

[A. V.]

Hi,

My boss wants to know how many events Flink can process, analyse etc. per second? I cant find this in the documentation.

[Andres Angel]

Hello A.V.

Id depends on the the underlying resources you are planing for your jobs. I mean memory and processing will play a principal role about this answer. keep in mind you are capable to break down your job in a number of parallel tasks by environment or even by an specific taks within your pipeline. 

I have scale my jobs up to around 1M per second without any trouble but keep in mind it depends as well on how complex look your pipeline likewise. Unlike the capabilities of parallel processing likely your pipeline applies enrichment and transformation reading data from an external ecosystem that can bring additional overload over your benchmark. Briefly, you should demo your scenario and get your own conclusions , but without any doubt Flink is capable of many stream processing records.

thanks 

AU 

On Wed, Oct 23, 2019 at 10:24 AM A. V. <[hidden email]> wrote:

Hi,

My boss wants to know how many events Flink can process, analyse etc. per second? I cant find this in the documentation.

[Michael Latta]

There are a lot of variables. How many cores are allocated, how much ram, etc. there are companies doing billions of events per day and more. 

Tell your boss it has proven to have extremely flat horizontal scaling. Meaning you can get it to process almost any number given sufficient hardware. 

You will need to do a proof of concept on your data and your analysis to know how much hardware is required for your problem. 

Michael

On Oct 23, 2019, at 7:24 AM, A. V. <[hidden email]> wrote:



Hi,

My boss wants to know how many events Flink can process, analyse etc. per second? I cant find this in the documentation.

[vino yang]

Hi A.V.

Add a few more points to the previous two answers. There is no clear answer to this question. In addition to resource issues, it depends on the size of the messages you are dealing with and the complexity of the logic. If you don't consider a lot of extra factors, look at the performance of the framework purely. You can go online to find some benchmark programs, test and cross-check. Here is a very popular benchmark[1] for stream computing, you can refer to it for testing.

Best,

Vino

[1]: https://github.com/yahoo/streaming-benchmarks

Michael Latta <[hidden email]> 于2019年10月23日周三 下午11:05写道：

There are a lot of variables. How many cores are allocated, how much ram, etc. there are companies doing billions of events per day and more. 

Tell your boss it has proven to have extremely flat horizontal scaling. Meaning you can get it to process almost any number given sufficient hardware. 

You will need to do a proof of concept on your data and your analysis to know how much hardware is required for your problem. 

Michael

On Oct 23, 2019, at 7:24 AM, A. V. <[hidden email]> wrote:



Hi,

My boss wants to know how many events Flink can process, analyse etc. per second? I cant find this in the documentation.