Hi Weihua,
> After dumping the memory and analyzing it, I found:
> Sink (121)'s RemoteInputChannel.unannouncedCredit = 0,
> Map (242)'s CreditBasedSequenceNumberingViewReader.numCreditsAvailable = 0.
> This is not consistent with my understanding of the Flink network transmission mechanism.
It probably is consistent. Downstream receiver unannounced all of the credits, and it’s simply waiting for the data to arrive, while upstream sender is waiting for the data to be sent down the stream.
Stack trace you posted confirms that the sink you posted has empty input buffer - it’s waiting for input data. Assuming rescale partitoning works as expected and indeed node 242 is connected to node 121, it implies the bottleneck is your data exchange between those two tasks. It could be
- network bottleneck (slow network? Packet losses?)
- machine swapping/long GC pauses (If upstream node is experiencing long pauses it might show up like this)
- cpu bottleneck in the network stack (frequent flushing? SSL?)
- some resource competition (too high parallelism for given number of machines)
- netty threads are not keeping up
It’s hard to say what’s the problem without looking at the resource usage (CPU/Network/Memory/Disk IO), GC logs, code profiling results.
Piotrek
PS Zhijiang:
RescalePartitioner in this case should be connect just two upstream subtasks with one downstream sink. Upstream subtasks N and N+1 should be connected to sink with N/2 id.
Hi, Zhijiang
I understand the normal credit-based backpressure mechanism. as usual the Sink inPoolUsage will be full, and the task stack will also have some information.
but this time is not the same. The Sink inPoolUsage is 0.
I also checked the stack. The Map is waiting org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestMemorySegment
The Sink is waiting data to deal, this is not very in line with expectations.
<粘贴的图形-2.tiff>
<粘贴的图形-1.tiff>
Hi Weihua,
From your below info, it is with the expectation in credit-based flow control.
I guess one of the sink parallelism causes the backpressure, so you will see that there are no available credits on Sink side and
the outPoolUsage of Map is almost 100%. It really reflects the credit-based states in the case of backpressure.
If you want to analyze the root cause of backpressure, you can trace the task stack of respective Sink parallelism to find which operation costs much,
then you can increase the parallelism or improve the UDF(if have bottleneck) to have a try. In addition, i am not sure why you choose rescale to shuffle data among operators. The default
forward mode can gain really good performance by default if you adjusting the same parallelism among them.
Best,
Zhijiang
------------------------------------------------------------------
Send Time:2020年5月24日(星期日) 18:32
Subject:Singal task backpressure problem with Credit-based Flow Control
Hi, all
I ran into a weird single Task BackPressure problem.
JobInfo:
DAG: Source (1000)-> Map (2000)-> Sink (1000), which is linked via rescale.
Flink version: 1.9.0
There is no related info in jobmanager/taskamanger log.
Through Metrics, I see that Map (242) 's outPoolUsage is full, but its downstream Sink (121)' s inPoolUsage is 0.
After dumping the memory and analyzing it, I found:
Sink (121)'s RemoteInputChannel.unannouncedCredit = 0,
Map (242)'s CreditBasedSequenceNumberingViewReader.numCreditsAvailable = 0.
This is not consistent with my understanding of the Flink network transmission mechanism.
Can someone help me? Thanks a lot.
Locked
