Programmatically abort checkpoint

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Programmatically abort checkpoint

Lorenzo Affetti-2
Hi everybody, I am using Flink v1.1.2

is it possible to programmatically abort a snapshot from the method

     public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp)

In an operator?

Thank you!

Lorenzo


Reply | Threaded
Open this post in threaded view
|

Re: Programmatically abort checkpoint

Stephan Ewen
What do you mean exactly with aborting a checkpoint? Continuing the processing despite failed checkpoints?

You can have a look at these recent changes, they cleanly abort checkpoint alignment in certain conditions: 


Best,
Stephan


On Fri, Nov 11, 2016 at 5:15 PM, Lorenzo Affetti <[hidden email]> wrote:
Hi everybody, I am using Flink v1.1.2

is it possible to programmatically abort a snapshot from the method

     public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp)

In an operator?

Thank you!

Lorenzo



Reply | Threaded
Open this post in threaded view
|

Re: Programmatically abort checkpoint

Lorenzo Affetti-2
Yes, I mean aborting the checkpoint alignment directly from an operator.
This is because I am introducing an operator that performs some asynchronous stuff that also involves side effects on its internal state.
I wanted to abort a checkpoint directly from that operator if a barrier comes in while I’m performing that asynchronous job.

Is the only way to use your code in master and send a DeclineCheckpoint message?

Thank you,

Lorenzo Affetti


On 11 Nov 2016, at 19:33, Stephan Ewen <[hidden email]> wrote:

What do you mean exactly with aborting a checkpoint? Continuing the processing despite failed checkpoints?

You can have a look at these recent changes, they cleanly abort checkpoint alignment in certain conditions: 


Best,
Stephan


On Fri, Nov 11, 2016 at 5:15 PM, Lorenzo Affetti <[hidden email]> wrote:
Hi everybody, I am using Flink v1.1.2

is it possible to programmatically abort a snapshot from the method

     public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp)

In an operator?

Thank you!

Lorenzo




Reply | Threaded
Open this post in threaded view
|

Re: Programmatically abort checkpoint

Ufuk Celebi
Hey Lorenzo,

internally Flink is able to abort checkpoints, but this is not possible from the user code. There is currently no way to be explicitly notified about an incoming barrier.

You can check out this PR (https://github.com/apache/flink/pull/2629) and see whether it addresses your question in another way. Looping in Till who reviewed the PR. Maybe he can answer this from the back of his head.

– Ufuk


On 11 November 2016 at 21:03:36, Lorenzo Affetti ([hidden email]) wrote:

> Yes, I mean aborting the checkpoint alignment directly from an operator.
> This is because I am introducing an operator that performs some asynchronous stuff that  
> also involves side effects on its internal state.
> I wanted to abort a checkpoint directly from that operator if a barrier comes in while  
> I’m performing that asynchronous job.
>  
> Is the only way to use your code in master and send a DeclineCheckpoint message?
>  
> Thank you,
>  
> Lorenzo Affetti
>  
>  
> On 11 Nov 2016, at 19:33, Stephan Ewen >  
> wrote:
>  
> What do you mean exactly with aborting a checkpoint? Continuing the processing despite  
> failed checkpoints?
>  
> You can have a look at these recent changes, they cleanly abort checkpoint alignment  
> in certain conditions:
>  
> https://issues.apache.org/jira/browse/FLINK-4976
> https://github.com/apache/flink/pull/2754
>  
> Best,
> Stephan
>  
>  
> On Fri, Nov 11, 2016 at 5:15 PM, Lorenzo Affetti >  
> wrote:
> Hi everybody, I am using Flink v1.1.2
>  
> is it possible to programmatically abort a snapshot from the method
>  
> public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp)  
>  
> In an operator?
>  
> Thank you!
>  
> Lorenzo
>  
>  
>  
>  
>  

Reply | Threaded
Open this post in threaded view
|

Re: Programmatically abort checkpoint

Lorenzo Affetti-2
Thank you for the pointers and the clarification!

One question, when you say: "There is currently no way to be explicitly notified about an incoming barrier”, isn’t `snapshotOperatorState` invoked when a barrier approaches the operator?

Lorenzo Affetti

-------
MD in computer engineering
PhD Student @ Politecnico di Milano
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)
Via Camillo Golgi 42, Milano (IT)


On 14 Nov 2016, at 10:03, Ufuk Celebi <[hidden email]> wrote:

There is currently no way to be explicitly notified about an incoming barrier

Reply | Threaded
Open this post in threaded view
|

Re: Programmatically abort checkpoint

Ufuk Celebi

On 14 November 2016 at 11:30:13, Lorenzo Affetti ([hidden email]) wrote:
> Thank you for the pointers and the clarification!
>  
> One question, when you say: "There is currently no way to be explicitly notified about  
> an incoming barrier”, isn’t `snapshotOperatorState` invoked when a barrier approaches  
> the operator?

Sorry for the confusion, I meant this in the context of keyed state. When you use the new CheckpointedFunction interface or the old Checkpointed interface, you indeed have the snapshot callbacks when the barrier arrives.

You can use this to notify the async threads, etc. but not explicitly cancel a checkpoint.

– Ufuk