Tools for Flink Job performance testing

[narasimha]

Hi, 

I'm new to the streaming world, checking on Performance testing tools.  Are there any recommended Performance testing tools for Flink? 

--

[Arvid Heise-3]

Hi,

performance testing is quite vague. Usually you start by writing a small first version of your pipeline and check how the well computation scales on your data. Flink's web UI [1] already helps quite well for the first time. Usually you'd also add some metric system and look for advanced metrics in there [2].

Now, you might be satisfied with your current solution and just start to extend it. Or you feel that it's not fast enough, scaling well enough. Then you can tweak your pipeline and perform smaller performance tests on your user code. Doing performance tests on the whole pipeline [3] would probably be hard if you just started as you also need to understand internals of Flink.

I also recommend to use Table API / SQL instead of DataStream if your application can be expressed well in relational operations. Table API already applies a wide range of optimizations that are much harder to implement manually in DataStream API. Table API will also bring your noticeable performance improvements over time when you update to a newer Flink version.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/back_pressure.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html

On Mon, Aug 10, 2020 at 1:06 PM narasimha <[hidden email]> wrote:

Hi, 

I'm new to the streaming world, checking on Performance testing tools.  Are there any recommended Performance testing tools for Flink? 

--

A.Narasimha Swamy

--

[narasimha]

Thanks, Arvid. 

The guide was helpful in how to start working with Flink. I'm currently exploring SQL/Table API.

Will surely come back for queries on it. 

On Thu, Aug 13, 2020 at 1:25 PM Arvid Heise <[hidden email]> wrote:

Hi,

performance testing is quite vague. Usually you start by writing a small first version of your pipeline and check how the well computation scales on your data. Flink's web UI [1] already helps quite well for the first time. Usually you'd also add some metric system and look for advanced metrics in there [2].

Now, you might be satisfied with your current solution and just start to extend it. Or you feel that it's not fast enough, scaling well enough. Then you can tweak your pipeline and perform smaller performance tests on your user code. Doing performance tests on the whole pipeline [3] would probably be hard if you just started as you also need to understand internals of Flink.

I also recommend to use Table API / SQL instead of DataStream if your application can be expressed well in relational operations. Table API already applies a wide range of optimizations that are much harder to implement manually in DataStream API. Table API will also bring your noticeable performance improvements over time when you update to a newer Flink version.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/back_pressure.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html

On Mon, Aug 10, 2020 at 1:06 PM narasimha <[hidden email]> wrote:

Hi, 

I'm new to the streaming world, checking on Performance testing tools.  Are there any recommended Performance testing tools for Flink? 

--

A.Narasimha Swamy

--

Arvid Heise | Senior Java Developer

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng   

--