Flink consuming rate increases slowly

[Eyal Pe'er]

Hi all,

Lately I've added more resources to my Flink cluster which required a restart of all apps.

From the cluster side, the only change I made, is to add more task slots.

On the cluster I have a streaming app that consumes from Kafka and sinks to files.

I noticed that since the restart, the applications "warmup" has impacted dramatically.

Before, the change it took few minutes for the app to start and consume normally (from my point of view, normally is a stable rate) - from 0 to 16K events per second in 4 minutes.

Now, after the change, it takes hours till it stabilizes on the normal processing rate- from 0 to 12K events per second in 3 hours.

The data source behavior hasn’t changed (same incoming rate, partitions, servers etc.).

I am aware to the backpressure mechanism in Flink, but it seems like it works too slow here.

Is there a way to speed or control it?

 

Thanks a lot

Eyal Peer

 

[Chen Qin]

Hi Eyal,

It’s unclear what warmup phase does in your use cases. Usually we see Flink start consume at high rate and drop to a point downstream can handle.

Thanks

Chen

On May 10, 2020, at 12:25 AM, Eyal Pe'er <[hidden email]> wrote:

Hi all,

Lately I've added more resources to my Flink cluster which required a restart of all apps.

From the cluster side, the only change I made, is to add more task slots.

On the cluster I have a streaming app that consumes from Kafka and sinks to files.

I noticed that since the restart, the applications "warmup" has impacted dramatically.

Before, the change it took few minutes for the app to start and consume normally (from my point of view, normally is a stable rate) - from 0 to 16K events per second in 4 minutes.

Now, after the change, it takes hours till it stabilizes on the normal processing rate- from 0 to 12K events per second in 3 hours.

The data source behavior hasn’t changed (same incoming rate, partitions, servers etc.).

I am aware to the backpressure mechanism in Flink, but it seems like it works too slow here.

Is there a way to speed or control it? 

 

Thanks a lot

Eyal Peer

[Dawid Wysakowicz-2]

Hi Eyal,

Could you explain your job a bit more? Did you increase the parallelism of your job? What does it do? Does it perform any time based operations? How do you measure the processing rate?

Best,

Dawid

On 10/05/2020 21:18, Chen Qin wrote:

Hi Eyal,

It’s unclear what warmup phase does in your use cases. Usually we see Flink start consume at high rate and drop to a point downstream can handle.

Thanks

Chen

On May 10, 2020, at 12:25 AM, Eyal Pe'er <[hidden email]> wrote:

Hi all,

Lately I've added more resources to my Flink cluster which required a restart of all apps.

From the cluster side, the only change I made, is to add more task slots.

On the cluster I have a streaming app that consumes from Kafka and sinks to files.

I noticed that since the restart, the applications "warmup" has impacted dramatically.

Before, the change it took few minutes for the app to start and consume normally (from my point of view, normally is a stable rate) - from 0 to 16K events per second in 4 minutes.

Now, after the change, it takes hours till it stabilizes on the normal processing rate- from 0 to 12K events per second in 3 hours.

The data source behavior hasn’t changed (same incoming rate, partitions, servers etc.).

I am aware to the backpressure mechanism in Flink, but it seems like it works too slow here.

Is there a way to speed or control it? 

 

Thanks a lot

Eyal Peer