Flink maxrecordcount increase causing a few task manager throughput drops

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink maxrecordcount increase causing a few task manager throughput drops

Terry Chia-Wei Wu
hi,

I change the following config from 
flink.shard.getrecords.maxrecordcount: 1000
flink.shard.getrecords.intervalmillis: 200

to 

flink.shard.getrecords.maxrecordcount: 10000
flink.shard.getrecords.intervalmillis: 1000

and found a few task managers around (10/1000) are becoming very slow. 
We also found that slow task manager's input stream shard (kineis) keeps aging. And other task manager's kinesis shard age stays low. 

Anyone knows why increase maxrecordcount and increase intervalmillis can cause a few task-managers throughput drops a lot? 

Best,
Terry

Reply | Threaded
Open this post in threaded view
|

Re: Flink maxrecordcount increase causing a few task manager throughput drops

rmetzger0
Hi Terry,

Why did you change the configuration?

It is indeed not intuitive for the throughput to drop. Maybe some internal throttling or rate limits on AWS side are causing this problem.

Best,
Robert


On Fri, Aug 7, 2020 at 10:48 AM Terry Chia-Wei Wu <[hidden email]> wrote:
hi,

I change the following config from 
flink.shard.getrecords.maxrecordcount: 1000
flink.shard.getrecords.intervalmillis: 200

to 

flink.shard.getrecords.maxrecordcount: 10000
flink.shard.getrecords.intervalmillis: 1000

and found a few task managers around (10/1000) are becoming very slow. 
We also found that slow task manager's input stream shard (kineis) keeps aging. And other task manager's kinesis shard age stays low. 

Anyone knows why increase maxrecordcount and increase intervalmillis can cause a few task-managers throughput drops a lot? 

Best,
Terry