Close method call in sinks

[Vieru, Mihail]

Hi,

quick question: is the close method in sinks, e.g. BucketingSink, called when a job gets cancelled or stopped?

We're trying to have the following: A custom stoppable streaming source with a modified BucketingSink which moves the files from ephemeral storage to S3 Frankfurt upon the closing of part files.

Best,
Mihail

[Till Rohrmann]

Hi Mihail,

the close method of the RichFunction will always be called. Only if the JVM crashes hard and the task thread does not have a chance to clean up the user functions, this won't be the case.

Cheers,

Till

On Fri, Mar 24, 2017 at 12:00 PM, Mihail Vieru <[hidden email]> wrote:

Hi,

quick question: is the close method in sinks, e.g. BucketingSink, called when a job gets cancelled or stopped?

We're trying to have the following: A custom stoppable streaming source with a modified BucketingSink which moves the files from ephemeral storage to S3 Frankfurt upon the closing of part files.

Best,
Mihail