Prevent job/operator from spilling to disk

[Maximilian Bode]

Hi everyone,

is there a way to prevent operators from spilling to disk? If not, would it be conceivable to make this configurable either per job or operator?

The use case is a batch application with the formal requirement not to persist in-flight data to disk (even temporarily) so it would be preferable to see the job fail and then be able to grant sufficient memory and run it again.

Cheers,

Max

— 

Maximilian Bode * Software Consultant * [hidden email]

TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring

Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke

Sitz: Unterföhring * Amtsgericht München * HRB 135082

[Fabian Hueske-2]

Hi Max,

it is not possible to deactivate spilling to disk at the moment.
It might be possible to implement, but this would require a few more changes to make it feasible.

For instance, we would need to add more fine-grained control about how memory is distributed among operators.
This is currently statically decided by the optimizer and there is no parameter to influence it.

Cheers, Fabian

2016-05-04 13:46 GMT+02:00 Maximilian Bode <[hidden email]>:

Hi everyone,

is there a way to prevent operators from spilling to disk? If not, would it be conceivable to make this configurable either per job or operator?

The use case is a batch application with the formal requirement not to persist in-flight data to disk (even temporarily) so it would be preferable to see the job fail and then be able to grant sufficient memory and run it again.

Cheers,

Max

— 

Maximilian Bode * Software Consultant * [hidden email]

TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring

Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke

Sitz: Unterföhring * Amtsgericht München * HRB 135082