Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)
Posted by
Bart Kastermans on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Stopping-a-kafka-consumer-gracefully-no-losing-of-inflight-events-StoppableFunction-tp18418p18479.html
Thanks for the reply; is there a flip for this?
- bart
On Mon, Feb 19, 2018, at 5:50 PM, Till Rohrmann wrote:
Hi Bart,
you're right that Flink currently does not support a graceful stop mechanism for the Kafka source. The community has already a good idea how to solve it in the general case and will hopefully soon add it to Flink.
Concerning the StoppableFunction: This interface was introduced quite some time ago and currently only works for some batch sources. In order to make it work with streaming, we need to add some more functionality to the engine in order to properly stop and take a savepoint.
Cheers,
Till
On Mon, Feb 19, 2018 at 3:36 PM, Bart Kastermans
<[hidden email]> wrote:
for gracefully stopping a job you need to implement the StoppableFunction interface. This
appears not (yet) implemented for Kafka consumers. Am I missing something, or is there a
different way to gracefully stop a job using a kafka source so we can restart it later without
losing any (in flight) events?
- bart