Diff between stop and cancel job

[Bajaj, Abhinav]

Hi,

Can some one please clarify the difference between stop and cancel of a Job.

Stop documentation mentions it is only for "streaming jobs only” but cancel also works for it.

Or can we resume a stopped streaming job ?

Thanks,

Abhi

Abhinav Bajaj Senior Engineer HERE Predictive Analytics Office:  [phone] Mobile: [phone]

HERE Seattle 701 Pike Street, #2000, Seattle, WA 98101, USA 47° 36' 41" N. 122° 19' 57" W HERE Maps

[rmetzger0]

Hi Abhi,

The difference between cancelling and stopping a (streaming) job is the following:

On a cancel call, the operators in a job immediately receive a `cancel()` method call to cancel them as
soon as possible.
If operators are not not stopping after the cancel call, Flink will start interrupting the thread periodically
until it stops.

A "stop" call is a more graceful way of stopping a running streaming job. Stop is only available for jobs
which use sources that implement the `StoppableFunction` interface. When the user requests to stop a job,
all sources will receive a `stop()` method call. The job will keep running until all sources properly shut down.
This allows the job to finish processing all inflight data.

In order to resume a job, you need to create a savepoint first. You can then stop or cancel the job and resume from the savepoint.

Regards,

Robert

On Thu, May 5, 2016 at 1:59 AM, Bajaj, Abhinav <[hidden email]> wrote:

Hi,

Can some one please clarify the difference between stop and cancel of a Job.

Stop documentation mentions it is only for "streaming jobs only” but cancel also works for it.

Or can we resume a stopped streaming job ?

Thanks,

Abhi

Abhinav Bajaj Senior Engineer HERE Predictive Analytics Office:  <a href="tel:%2B12062092767" value="[phone]" target="_blank">[phone] Mobile: <a href="tel:%2B17083299516" value="[phone]" target="_blank">[phone]

HERE Seattle 701 Pike Street, #2000, Seattle, WA 98101, USA 47° 36' 41" N. 122° 19' 57" W HERE Maps

[Ufuk Celebi]

On Thu, May 5, 2016 at 1:59 AM, Bajaj, Abhinav <[hidden email]> wrote:

Or can we resume a stopped streaming job ?

You can use savepoints [1] to take a snapshot of a streaming program from which you can restart the job at a later point in time. This is independent of whether you cancel or stop the program after taking the savepoint.

Currently, you can effectively only use savepoint + cancel.

Stopping is currently only implemented for the TwitterSource. I expect this to be implemented for other more common sources like Kafka as well in future versions. Futhermore, I expect a convenience feature like stop-and-savepoint, which  gracefully stops your streaming program and automatically takes a savepoint. But as of now, I probably need to go with savepoint + cancel.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/savepoints.html