How back pressure is handled by source?

[Pawel Bartoszek]

Hi,

I am interested in how back pressure is handled by sources in Flink ie Kinesis source. From what I understood back pressure is a mechanism to slow down rate at which records are read from the stream. However, in the kinesis source code I can see that it configured to read the same number of rows (default set by aws sdk kinesis library).   

[Stefan Richter]

Hi,

backpressure comes into play when the source is attempting to pass the generated events to downstream operators. If the downstream operators build up backpressure, passing data to them can block. You might think of this like a bounded queue that is full in case of backpressure and blocks until capacity is again available, so the source is slowed down because it will have to wait until it becomes unblocked before the loop comes again to the „reading" part. Maybe this is helpful: https://data-artisans.com/blog/how-flink-handles-backpressure .

Best,

Stefan

Am 30.01.2018 um 14:36 schrieb Pawel Bartoszek <[hidden email]>:

Hi,

I am interested in how back pressure is handled by sources in Flink ie Kinesis source. From what I understood back pressure is a mechanism to slow down rate at which records are read from the stream. However, in the kinesis source code I can see that it configured to read the same number of rows (default set by aws sdk kinesis library).