Does Flink automatically apply any backpressure ?

[Suraj Puvvada]

Hello

I am trying to understand if Flink has a mechanism to automatically apply any backpressure by throttling any operators ? For example if I have a Process function that reads from a Kafkaa source and writes to a Kafka sink. If the process function is slow will the kafka source be automatically throttled ?

Thanks

Suraj  

[Jake]

Hi Suraj Puvvada

Yes, Flink back pressure depend on the Flink task buffer。process task will sends buffer remaining size to source, source will slow down.

https://www.ververica.com/blog/how-flink-handles-backpressure

Jake

On Jul 31, 2020, at 2:48 AM, Suraj Puvvada <[hidden email]> wrote:

Hello

I am trying to understand if Flink has a mechanism to automatically apply any backpressure by throttling any operators ? For example if I have a Process function that reads from a Kafkaa source and writes to a Kafka sink. If the process function is slow will the kafka source be automatically throttled ?

Thanks

Suraj  

[Danny Chan]

Yes, just like Jake said, the back pressure happened automatically and usually there is no need to tweak it, you actually can have configure the metrics about it, see [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/back_pressure.html

Best,

Danny Chan

在 2020年7月31日 +0800 AM10:28，Jake <[hidden email]>，写道：

Hi Suraj Puvvada

Yes, Flink back pressure depend on the Flink task buffer。process task will sends buffer remaining size to source, source will slow down.

https://www.ververica.com/blog/how-flink-handles-backpressure

Jake

On Jul 31, 2020, at 2:48 AM, Suraj Puvvada <[hidden email]> wrote:

Hello

I am trying to understand if Flink has a mechanism to automatically apply any backpressure by throttling any operators ? For example if I have a Process function that reads from a Kafkaa source and writes to a Kafka sink. If the process function is slow will the kafka source be automatically throttled ?

Thanks

Suraj