Async and checkpointing

[anurag]

Hi All,

Thanks for your help in advance. I am using async I/O with  flink 1.5.5, I am using

AsyncDataStream.unorderedWait method  also my capacity is set to 100.My question is since my capacity is 100, each thread will be processing one record.Say sequence number of my records is S1,S2....S100 and say at a particular point in time , thread T1 is processing record S1, T2 is processing record S2  and T100 is processing record S100 and say all the threads T1..T100 except T50 failed when sending the record to the Sink. In this will the checkpoint pointer move at all or not.

Apologies in advance if my question is not clear.

Thanks,
Anurag

[Yun Tang]

Hi  Anurag

What do you mean "will the checkpoint pointer move at all or not"?

If one of your thread failed to send record, and if it would cause that sub-task to fail, it would lead to the job failover. When job failover, any on-going checkpoint would be aborted and job would then just restore from last latest checkpoint.

If failing to send record would not cause that sub-task to fail, nothing would happen and job continues to run but this might be not what you want.

Best

Yun Tang

---

From: anurag <[hidden email]>
Sent: Friday, October 11, 2019 2:03
To: [hidden email] <[hidden email]>
Subject: Async and checkpointing

 

Hi All,

Thanks for your help in advance. I am using async I/O with  flink 1.5.5, I am using

AsyncDataStream.unorderedWait method  also my capacity is set to 100.My question is since my capacity is 100, each thread will be processing one record.Say sequence number of my records is S1,S2....S100 and say at a particular point in time , thread T1 is processing record S1, T2 is processing record S2  and T100 is processing record S100 and say all the threads T1..T100 except T50 failed when sending the record to the Sink. In this will the checkpoint pointer move at all or not.

Apologies in advance if my question is not clear.

Thanks,
Anurag

[anurag]

Hi Yun,

Thanks for your reply. I am sorry if I was not clear . What I meant was that as records are processed, the checkpoint and the corresponding stream position will move.

But in case of async with unordered, will the checkpoint and the corresponding stream position move in the above scenario where Tasks T1....T100 are currently running in parallel and all tasks T1...T100 pass except for T50.

Many thanks.

Anurag

On Thu, Oct 10, 2019 at 11:13 AM Yun Tang <[hidden email]> wrote:

Hi  Anurag

What do you mean "will the checkpoint pointer move at all or not"?

If one of your thread failed to send record, and if it would cause that sub-task to fail, it would lead to the job failover. When job failover, any on-going checkpoint would be aborted and job would then just restore from last latest checkpoint.

If failing to send record would not cause that sub-task to fail, nothing would happen and job continues to run but this might be not what you want.

Best

Yun Tang

---

From: anurag <[hidden email]>
Sent: Friday, October 11, 2019 2:03
To: [hidden email] <[hidden email]>
Subject: Async and checkpointing

 

Hi All,

Thanks for your help in advance. I am using async I/O with  flink 1.5.5, I am using

AsyncDataStream.unorderedWait method  also my capacity is set to 100.My question is since my capacity is 100, each thread will be processing one record.Say sequence number of my records is S1,S2....S100 and say at a particular point in time , thread T1 is processing record S1, T2 is processing record S2  and T100 is processing record S100 and say all the threads T1..T100 except T50 failed when sending the record to the Sink. In this will the checkpoint pointer move at all or not.

Apologies in advance if my question is not clear.

Thanks,
Anurag