Why would slot sharing be undesirable?

[leon_mclare]

Hi,

i am deploying a 5 stage pipeline, with varying DOPs for experimental purposes. The benefits of slot sharing are clear. What is unclear to me is when slot sharing would be undesirable.

Since slot sharing enables bypassing of additional buffers (buffers between slots) as well as memory based optimizations, it surely is always beneficial. Furthermore, since all slots run within the same JVM, there also appears to be no advantage of disabling slot sharing in regard to simplifying JVM testing/debugging or for other isolation purposes.

Could you perhaps elaborate on when slot sharing may be undesirable?

Regards
Leon

[Márton Balassi]

Hi Leon,

Basically you are trading away utilizing all of your resources in the cluster to cut network IO.

With putting more and more tasks into the same slot you are pushing a job from being potentially network bound to usually CPU bound, as the parallel instances of tasks in the same slot sharing group are scheduled on the same machine.

Have a look at the basic metrics of the working nodes of the cluster to determine when to stop sharing.

Best,

Marton

On Sat, Jun 4, 2016 at 10:09 AM, <[hidden email]> wrote:

Hi,

i am deploying a 5 stage pipeline, with varying DOPs for experimental purposes. The benefits of slot sharing are clear. What is unclear to me is when slot sharing would be undesirable.

Since slot sharing enables bypassing of additional buffers (buffers between slots) as well as memory based optimizations, it surely is always beneficial. Furthermore, since all slots run within the same JVM, there also appears to be no advantage of disabling slot sharing in regard to simplifying JVM testing/debugging or for other isolation purposes.

Could you perhaps elaborate on when slot sharing may be undesirable?

Regards
Leon