(DEPRECATED) Apache Flink User Mailing List archive.

Re: How to measure Flink performance

Posted by Ken Krugler on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/How-to-measure-Flink-performance-tp6741p6910.html

Hi Dhruv,

On May 12, 2016, at 11:07pm, Dhruv Gohil <[hidden email]> wrote:

Hi Prateek,

https://github.com/dataArtisans/yahoo-streaming-benchmark/blob/master/flink-benchmarks/src/main/java/flink/benchmark/utils/ThroughputLogger.java
https://github.com/dataArtisans/yahoo-streaming-benchmark/blob/master/flink-benchmarks/src/main/java/flink/benchmark/utils/AnalyzeTool.java

help you measure throughput and latency both, from within topology.

The AnalyzeTool is processing a file that has lines which match the pattern:

Pattern latencyPattern = Pattern.compile(".*Latency ([0-9]+) ms.*”);

This isn’t something created by the ThroughputLogger.

Who generates these files, and how do they calculate the latency?

Thanks,

— Ken

On Thursday 12 May 2016 11:21 PM, Konstantin Knauf wrote:

Hi Prateek,

regarding throughput, what about simply filling the input Kafka topic
with some (a lot) of messages and monitor (e.g.
http://quantifind.github.io/KafkaOffsetMonitor/) how quickly Flink can
work the lag off. The messages should be representative of your use
case, of course.

Latency is harder, I think, and I would also be interested in the
approaches of others to measure latency in Flink.

To some extend, you could do it by adding some logging inside Flink, but
this effects latency and only measure latency whithin Flink (excluding
reading from source and writing to sink).

Cheers,

Konstantin

On 12.05.2016 18:57, prateekarora wrote:

Hi

How can i measure  throughput and latency  of my application in flink 1.0.2
?

Regards
Prateek



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-measure-Flink-performance-tp6741p6863.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

--------------------------

Ken Krugler

+1 530-210-6378

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr