Question regarding configuring number of network buffers

[Ray Ruvinskiy]

The documentation provides the formula #slots-per-TM^2 * #TMs * 4 to determine the number of network buffers we should configure. The documentation also says, “A logical network connection exists for each point-to-point exchange of data over the network, which typically happens at repartitioning- or broadcasting steps (shuffle phase). In those, each parallel task inside the TaskManager has to be able to talk to all other parallel tasks.” Does that say that each parallel task inside the TaskManager talk to all parallel tasks inside the same TaskManager or to all parallel tasks across all task managers? Intuitively, I would assume the latter, but then wouldn’t the formula for determining the number of network buffers be more along the lines of (#slots-per-TM * #TMs)^2?

 

Thanks,

 

Ray

 

[Tzu-Li (Gordon) Tai]

Hi!

I’ve looped in Nico (cc’ed) who knows about the configuration of network buffers the best, and could probably explain it to you in more detail.

Cheers,

Gordon

On 7 June 2017 at 5:59:08 PM, Ray Ruvinskiy ([hidden email]) wrote:

The documentation provides the formula #slots-per-TM^2 * #TMs * 4 to determine the number of network buffers we should configure. The documentation also says, “A logical network connection exists for each point-to-point exchange of data over the network, which typically happens at repartitioning- or broadcasting steps (shuffle phase). In those, each parallel task inside the TaskManager has to be able to talk to all other parallel tasks.” Does that say that each parallel task inside the TaskManager talk to all parallel tasks inside the same TaskManager or to all parallel tasks across all task managers? Intuitively, I would assume the latter, but then wouldn’t the formula for determining the number of network buffers be more along the lines of (#slots-per-TM * #TMs)^2?

 

Thanks,

 

Ray