Hi everybody, I am using Flink v1.1.2
is it possible to programmatically abort a snapshot from the method
public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp)
In an operator?
Thank you!
Lorenzo |
What do you mean exactly with aborting a checkpoint? Continuing the processing despite failed checkpoints?
You can have a look at these recent changes, they cleanly abort checkpoint alignment in certain conditions: Best, Stephan On Fri, Nov 11, 2016 at 5:15 PM, Lorenzo Affetti <[hidden email]> wrote:
|
Yes, I mean aborting the checkpoint alignment directly from an operator.
This is because I am introducing an operator that performs some asynchronous stuff that also involves side effects on its internal state.
I wanted to abort a checkpoint directly from that operator if a barrier comes in while I’m performing that asynchronous job.
Is the only way to use your code in master and send a DeclineCheckpoint message?
Thank you,
Lorenzo Affetti
|
Hey Lorenzo,
internally Flink is able to abort checkpoints, but this is not possible from the user code. There is currently no way to be explicitly notified about an incoming barrier. You can check out this PR (https://github.com/apache/flink/pull/2629) and see whether it addresses your question in another way. Looping in Till who reviewed the PR. Maybe he can answer this from the back of his head. – Ufuk On 11 November 2016 at 21:03:36, Lorenzo Affetti ([hidden email]) wrote: > Yes, I mean aborting the checkpoint alignment directly from an operator. > This is because I am introducing an operator that performs some asynchronous stuff that > also involves side effects on its internal state. > I wanted to abort a checkpoint directly from that operator if a barrier comes in while > I’m performing that asynchronous job. > > Is the only way to use your code in master and send a DeclineCheckpoint message? > > Thank you, > > Lorenzo Affetti > > > On 11 Nov 2016, at 19:33, Stephan Ewen > > wrote: > > What do you mean exactly with aborting a checkpoint? Continuing the processing despite > failed checkpoints? > > You can have a look at these recent changes, they cleanly abort checkpoint alignment > in certain conditions: > > https://issues.apache.org/jira/browse/FLINK-4976 > https://github.com/apache/flink/pull/2754 > > Best, > Stephan > > > On Fri, Nov 11, 2016 at 5:15 PM, Lorenzo Affetti > > wrote: > Hi everybody, I am using Flink v1.1.2 > > is it possible to programmatically abort a snapshot from the method > > public StreamTaskState snapshotOperatorState(long checkpointId, long timestamp) > > In an operator? > > Thank you! > > Lorenzo > > > > > |
Thank you for the pointers and the clarification!
One question, when you say: "There is currently no way to be explicitly notified about an incoming barrier”, isn’t `snapshotOperatorState` invoked when a barrier approaches the operator?
Lorenzo Affetti
-------
MD in computer engineering PhD Student @ Politecnico di Milano Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)
Via Camillo Golgi 42, Milano (IT)
|
On 14 November 2016 at 11:30:13, Lorenzo Affetti ([hidden email]) wrote: > Thank you for the pointers and the clarification! > > One question, when you say: "There is currently no way to be explicitly notified about > an incoming barrier”, isn’t `snapshotOperatorState` invoked when a barrier approaches > the operator? Sorry for the confusion, I meant this in the context of keyed state. When you use the new CheckpointedFunction interface or the old Checkpointed interface, you indeed have the snapshot callbacks when the barrier arrives. You can use this to notify the async threads, etc. but not explicitly cancel a checkpoint. – Ufuk |
Free forum by Nabble | Edit this page |