End to End Performance Testing

[WILSON Frank]

Hi,

 

I am looking for resources to help me test the performance of my Flink pipelines end-to-end. I want to verify that my pipelines meet throughput and latency requirements (so for a given number of events per second the latency of the output is under so many seconds). I read that Alibaba had developed a Blink test platform that offers performance and stability testing [1] that seem to be relevant to this problem.

 

I read also that Blink has been contributed back to Flink [2] and I was wondering if the test platform is also included in the branch [3]?

 

Thanks,

 

 

Frank Wilson

 

 

[1] https://hackernoon.com/from-code-quality-to-integration-optimizing-alibabas-blink-testing-framework-dc9c357319de#190e

 

[2] https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html

 

[3] https://github.com/apache/flink/tree/blink

 

[Yu Li]

A short answer is no, the test platform is not included in the blink branch.

FWIW (to make it clear, I'm not the decision maker so just some of my own opinions), the test platform is for production usage and includes simulation of online jobs and probably some confidential stuff, so I don't think it could be rawly put out. However, there're indeed ongoing plan to extract test cases, benchmarks, etc. from the platform and contribute to Flink community, to enrich the Flink test suite. Please watch the news and discussions in flink mailing list for the progress (smile).

On Tue, 2 Apr 2019 at 18:49, WILSON Frank <[hidden email]> wrote:

Hi,

 

I am looking for resources to help me test the performance of my Flink pipelines end-to-end. I want to verify that my pipelines meet throughput and latency requirements (so for a given number of events per second the latency of the output is under so many seconds). I read that Alibaba had developed a Blink test platform that offers performance and stability testing [1] that seem to be relevant to this problem.

 

I read also that Blink has been contributed back to Flink [2] and I was wondering if the test platform is also included in the branch [3]?

 

Thanks,

 

 

Frank Wilson

 

 

[1] https://hackernoon.com/from-code-quality-to-integration-optimizing-alibabas-blink-testing-framework-dc9c357319de#190e

 

[2] https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html

 

[3] https://github.com/apache/flink/tree/blink