Checkpointing when one of the sources has completed

[Niels van Kaam]

Hi All,

I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.

Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?

Cheers,

Niels

[Paul Lam]

Hi Niels,

Please see https://issues.apache.org/jira/browse/FLINK-249.

Best,

Paul Lam

在 2018年10月17日，16:58，Niels van Kaam <[hidden email]> 写道：

Hi All,

I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.

Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?

Cheers,

Niels

[Fabian Hueske-2]

Hi Niels,

Checkpoints can only complete if all sources are running.

That's because the checkpoint mechanism relies on injecting checkpoint barriers into the stream at the sources.

Best, Fabian

Am Mi., 17. Okt. 2018 um 11:11 Uhr schrieb Paul Lam <[hidden email]>:

Hi Niels,

Please see https://issues.apache.org/jira/browse/FLINK-249.

Best,

Paul Lam

在 2018年10月17日，16:58，Niels van Kaam <[hidden email]> 写道：

Hi All,

I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.

Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?

Cheers,

Niels

[Joshua Fan]

Hi Niels,

Probably not, an operator begins to do checkpoint until it gets all the barriers from all the upstream sources, if one source can not send a barrier, the downstream operator can not do checkpoint, FYI.

Yours sincerely

Joshua

On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam <[hidden email]> wrote:

Hi All,

I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.

Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?

Cheers,

Niels

[Niels van Kaam]

Hi All,

Thanks for the responses, the finished source explains my issue then. I can work around the problem by letting my sources negotiate a "final" checkpoint via zookeeper.

@Paul, I think your answer was meant for the earlier question asked by Joshua?

Cheers,

Niels

On Wed, Oct 17, 2018 at 11:15 AM Joshua Fan <[hidden email]> wrote:

Hi Niels,

Probably not, an operator begins to do checkpoint until it gets all the barriers from all the upstream sources, if one source can not send a barrier, the downstream operator can not do checkpoint, FYI.

Yours sincerely

Joshua

On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam <[hidden email]> wrote:

Hi All,

I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.

Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?

Cheers,

Niels

[Paul Lam]

Hi Niels,

The link was broken, it should be  https://issues.apache.org/jira/browse/FLINK-2491.

A similar question was asked a few days ago.

Best,

Paul Lam

在 2018年10月17日，19:56，Niels van Kaam <[hidden email]> 写道：

Hi All,

Thanks for the responses, the finished source explains my issue then. I can work around the problem by letting my sources negotiate a "final" checkpoint via zookeeper.

@Paul, I think your answer was meant for the earlier question asked by Joshua?

Cheers,

Niels

On Wed, Oct 17, 2018 at 11:15 AM Joshua Fan <[hidden email]> wrote:

Hi Niels,

Probably not, an operator begins to do checkpoint until it gets all the barriers from all the upstream sources, if one source can not send a barrier, the downstream operator can not do checkpoint, FYI.

Yours sincerely

Joshua

On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam <[hidden email]> wrote:

Hi All,

I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output.

Does anyone know if Flink's periodic checkpoints are supposed to continue when one or more sources of a job are in the "FINISHED" state?

Cheers,

Niels