What happens to the channels when there is backpressure?

[Felipe Gutierrez]

Hi community,

I have a question about backpressure. Suppose a scenario that I have a map and a reducer, and the reducer is back pressuring the map operator. I know that the reducer is processing tuples at a lower rate than it is receiving.

However, can I say that at least one channel between the map and the reducer is totally using its available bandwidth?

My guess is it is not, at least in the beginning. But as the time goes on the tuples will be queued in the network buffer of the reducer and then the bandwidth will be 100% of usage. Am I right?

Thanks,

Felipe

[yingjie]

Hi Felipe,

That depends on what do you mean by 'bandwidth'. If you mean the capability of the network stack, the answer would be no.

Here is a post about Flink network stack which may help: https://flink.apache.org/2019/06/05/flink-network-stack.html.

Thanks,

Yingjie

Felipe Gutierrez <[hidden email]> 于2019年11月27日周三 下午11:13写道：

Hi community,

I have a question about backpressure. Suppose a scenario that I have a map and a reducer, and the reducer is back pressuring the map operator. I know that the reducer is processing tuples at a lower rate than it is receiving.

However, can I say that at least one channel between the map and the reducer is totally using its available bandwidth?

My guess is it is not, at least in the beginning. But as the time goes on the tuples will be queued in the network buffer of the reducer and then the bandwidth will be 100% of usage. Am I right?

Thanks,

Felipe

[Felipe Gutierrez]

Hi Yingjie,

I read this post and the next one as well (https://flink.apache.org/2019/07/23/flink-network-stack-2.html).

I mean the bandwidth of the channels between two physical operators. When they are in different hosts, so when the channels are a network channel.

Thanks

On Thu, Nov 28, 2019 at 5:07 AM yingjie cao <[hidden email]> wrote:

Hi Felipe,

That depends on what do you mean by 'bandwidth'. If you mean the capability of the network stack, the answer would be no.

Here is a post about Flink network stack which may help: https://flink.apache.org/2019/06/05/flink-network-stack.html.

Thanks,

Yingjie

Felipe Gutierrez <[hidden email]> 于2019年11月27日周三 下午11:13写道：

Hi community,

I have a question about backpressure. Suppose a scenario that I have a map and a reducer, and the reducer is back pressuring the map operator. I know that the reducer is processing tuples at a lower rate than it is receiving.

However, can I say that at least one channel between the map and the reducer is totally using its available bandwidth?

My guess is it is not, at least in the beginning. But as the time goes on the tuples will be queued in the network buffer of the reducer and then the bandwidth will be 100% of usage. Am I right?

Thanks,

Felipe