Insufficient number of network buffers- what does Total mean on the Flink Dashboard

Posted by Vijay Balakrishnan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Insufficient-number-of-network-buffers-what-does-Total-mean-on-the-Flink-Dashboard-tp35901.html

Hi,
Get this error:
java.io.IOException: Insufficient number of network buffers: required 2, but only 0 available. The total number of network buffers is currently set to 877118 of 32768 bytes each. You can increase this number by setting the configuration keys 'taskmanager.network.memory.fraction', 'taskmanager.network.memory.min', and 'taskmanager.network.memory.max'.
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/dispatcher#-1420732632]] after [10000 ms]. Message of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.


Followed docs here:
https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/mem_setup.html

network = Min(max, Max(min, fraction x total)  //what does Total mean - The max JVM heap is used to derive the total memory for the calculation of network buffers. - can I see it in the Flink Dashboard ??? 117GB here ?
= Min(50G, Max(500mb, Max(0.48 * 117G))  ) = MIn(50G, 56.16G)= 50G
877118 of 32768 bytes each comes to 28.75GB. So, why is it failing ?
Used this in flink-conf.yaml:
    taskmanager.numberOfTaskSlots: 10
    rest.server.max-content-length: 314572800
    taskmanager.network.memory.fraction: 0.45
    taskmanager.network.memory.max: 50gb
    taskmanager.network.memory.min: 500mb
    akka.ask.timeout: 240s
    cluster.evenly-spread-out-slots: true
    akka.tcp.timeout: 240s
taskmanager.network.request-backoff.initial: 5000
taskmanager.network.request-backoff.max: 30000
web.timeout:1000000
web.refresh-interval:6000

Saw some old calc about buffers
(slots/Tm * slots/TM) * #TMs * 4
=10 * 10 * 47 * 4 = 18,800 buffers.

What am I missing in the network buffer calc ??  

TIA,