Not enough buffers when running Flink jar on cluster

[Hung]

Hi,

I have a question regarding not enough buffers when running Flink jar on cluster.

The data set is a graph and the arcs file is around 65GB and the algorithms are top 10 out-degree and in-degree

When running jar it gives an error "System has not enough buffers to execute tasks".
and found this issue in Apache Flink's documentation:

taskmanager.network.numberOfBuffers: The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: 2048).

The confusing thing is that others also run some algorithms with similar size dataset and complexity using the same cluster. Can increase buffer really solve this issue?

[rmetzger0]

Hi,

the number of required buffers depends a lot on the job. If a job needs to send data to operators running on other machines, it needs buffers to do that.
Operations like reduce or join usually need many buffers. If you just have a bunch of chained mappers, the requirements are lower.

Increasing the number of buffers will really solve this problem.