How to explain the latency at different source injection rate?

[varuy322]

Hi there,

I have got confused by the issue about the end-2-end latency statistic. When I run benchmark(similar to yahoo streaming benchmark, yet without Redis, data generated by Flink, sink to Kafka),
key parameters set as below:
  Task manager(short for tm) number:  3,
  slots per tm:  3,
  parallel:  9,
  buffer. Timeout:  100ms

the plan as below:
[cid:[e-mail]77F590]

When I set injection rate to be 10k, the latency measured is 108ms, but when set injection rate to be 1k, latency is 560ms.
[cid:[e-mail]77F590]

IMO, the low injection rate may get low latency, but the result couldn't match. Some body could give me the explain, thanks.

Best Regards & Thanks

Rui



image003.jpg (17K) <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/attachment/31405/0/image003.jpg>
image004.jpg (90K) <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/attachment/31405/1/image004.jpg>

[Till Rohrmann]

Hi Rui,

it is hard to explain the results you are observing without knowing the complete benchmark setup. For example, it would be interesting to know the exact details of your job and the way you are measuring latencies. W/o this information I would only be able to guess things.

Cheers,

Till

On Wed, Dec 4, 2019 at 7:04 AM Wang, Rui2 <[hidden email]> wrote:

Hi there,

 

I have got confused by the issue about the end-2-end latency statics. When I run benchmark(similar to yahoo streaming benchmark, yet without Redis, data generated by Flink, sink to Kafka),

key parameters set as below:

  Task manager(short for tm) number:  3,

  slots per tm:  3,  

  parallel:  9,

  buffer. Timeout:  100ms

 

the plan as below:

 

When I set injection rate to be 10k, the latency measured is 108ms, but when set injection rate to be 1k, latency is 560ms.

 

IMO, the low injection rate may get low latency, but the result could match. Some body could give me the explain, thanks.

 

Best Regards & Thanks

 

Rui