Back pressure within a operator chain

[Paul Lam]

Hi,

I would like to know if back pressure applies to operators in the same operator chain?

The background is that I have a simple streaming job that consumes data from Kafka, do some transformation and writes to HDFS (all the operators are chained together), and if the Kafka partitions are much greater that job parallelism (like 40:1), OOM happens. The the root cause should be Kafka consumer pulling too much data. So I’m wondering if I should separate the source and sink to make the back pressure mechanism working.

Best,

Paul Lam

[Chesnay Schepler]

The behavior should be identical regardless of whether the are chained or not.

On 23.01.2019 09:11, Paul Lam wrote:

Hi,

I would like to know if back pressure applies to operators in the same operator chain?

The background is that I have a simple streaming job that consumes data from Kafka, do some transformation and writes to HDFS (all the operators are chained together), and if the Kafka partitions are much greater that job parallelism (like 40:1), OOM happens. The the root cause should be Kafka consumer pulling too much data. So I’m wondering if I should separate the source and sink to make the back pressure mechanism working.

Best,

Paul Lam