Re: How graceful shutdown or resource clean up happens in Flink at task level ?

Posted by Stefan Richter on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/How-graceful-shutdown-or-resource-clean-up-happens-in-Flink-at-task-level-tp16825p16836.html

Hi,


the user function’s close() method is called in AbstractStreamOperator::close() and ::dispose(). The invocation of the user function’s close() in AbstractStreamOperator::dispose() only has an effect if there was no previous invocation of the method through AbstractStreamOperator::close().

AbstractStreamOperator::close() and ::dispose(), in turn, are called inside StreamTask::invoke(), which also runs the operator’s main processing loop (AbstractStreamOperator::run()). AbstractStreamOperator::close() happens through closeAllOperators(), after AbstractStreamOperator::run(). In case that run() is exited through a Throwable, we end up in a catch block for Throwable that invokes AbstractStreamOperator::dispose(). So the UDF is either closed normally, after the operator’s run method ended or exceptional through the operator’s dispose() method.

For your 3 cases this means:

> 1. Some programmatic error (NullPointer Exception)

Will end up in the catch-block around the operator’s run() method and reach UDF’s close() via AbstractStreamOperator::dispose().

> 2. OutOfMemoryError

Same as for (1.), but there is no strict guarantee that we have enough heap memory to actually perform the close() call.

> 3. Syste.exit(0)

This terminates the JVM immediately, no further code will be executed and therefore no cleanup can happen.

Best,
Stefan