granular configuration of cluster

[Stefano Bortoli]

Hi all,

is it possible to configure a cluster with different number of task manager per node?

for example, I have a cluster where I am using a global index, and the nodes where the global index in running should not be overloaded. However, the other nodes could run more than 4 task manager per node.

To have such features would be very helpful to fine tune processes on small clusters.

thanks a lot for your support!

saluti,
Stefano

[Stephan Ewen]

Sure that's possible. You need different configurations for each node, so instead of having one config directory shared across all nodes, give each node an individual config.

The nodes that should do more work need to see a config with a larger JVM heap and more task slots, the nodes that should have fewer load need to have a config with smaller heap and fewer slots.

On Fri, Nov 28, 2014 at 4:57 PM, Stefano Bortoli <[hidden email]> wrote:

Hi all,

is it possible to configure a cluster with different number of task manager per node?

for example, I have a cluster where I am using a global index, and the nodes where the global index in running should not be overloaded. However, the other nodes could run more than 4 task manager per node.

To have such features would be very helpful to fine tune processes on small clusters.

thanks a lot for your support!

saluti,
Stefano

[Stefano Bortoli]

sweet! so it does not have to be consistent across all nodes! then each taskmanager will report to the job manager, that will understand the distribution of the work accordingly. I guess.

thanks a lot!

2014-11-28 17:12 GMT+01:00 Stephan Ewen <[hidden email]>:

Sure that's possible. You need different configurations for each node, so instead of having one config directory shared across all nodes, give each node an individual config.

The nodes that should do more work need to see a config with a larger JVM heap and more task slots, the nodes that should have fewer load need to have a config with smaller heap and fewer slots.

On Fri, Nov 28, 2014 at 4:57 PM, Stefano Bortoli <[hidden email]> wrote:

Hi all,

is it possible to configure a cluster with different number of task manager per node?

for example, I have a cluster where I am using a global index, and the nodes where the global index in running should not be overloaded. However, the other nodes could run more than 4 task manager per node.

To have such features would be very helpful to fine tune processes on small clusters.

thanks a lot for your support!

saluti,
Stefano

[Stephan Ewen]

Yes, each task manager offers a certain number of "task slots" to the JobManager. That is what you see in the web frontend also. Each TaskManager can offer a different number of slots, there is no restriction there.

Each slot takes an operator (or a pipeline of operators). So, if a task manager offers 4 slots, it can execute 4 parallel mappers, reducers, ... of the same type concurrently. A task manager that offers 8 slots can execute 8 concurrently, ...

On Fri, Nov 28, 2014 at 5:40 PM, Stefano Bortoli <[hidden email]> wrote:

sweet! so it does not have to be consistent across all nodes! then each taskmanager will report to the job manager, that will understand the distribution of the work accordingly. I guess.

thanks a lot!

2014-11-28 17:12 GMT+01:00 Stephan Ewen <[hidden email]>:

Sure that's possible. You need different configurations for each node, so instead of having one config directory shared across all nodes, give each node an individual config.

The nodes that should do more work need to see a config with a larger JVM heap and more task slots, the nodes that should have fewer load need to have a config with smaller heap and fewer slots.

On Fri, Nov 28, 2014 at 4:57 PM, Stefano Bortoli <[hidden email]> wrote:

Hi all,

is it possible to configure a cluster with different number of task manager per node?

for example, I have a cluster where I am using a global index, and the nodes where the global index in running should not be overloaded. However, the other nodes could run more than 4 task manager per node.

To have such features would be very helpful to fine tune processes on small clusters.

thanks a lot for your support!

saluti,
Stefano