reactive mode and back pressure

[Alexey Trenikhun]

Hello,

Is new reactive mode can operate under back pressure? Old manual rescaling via taking savepoint didn't work with system under back pressure, since it was practically impossible to take savepoint, so wondering is reactive mode expected to be better in this regards ?

Thanks,

Alexey 

[Xintong Song]

Hi Alexey,

I don't think the new reactive mode makes any changes to the checkpoint/savepoint mechanism, at least not at the moment.

However, you might want to take a look at the unaligned checkpoint [1]. The unaligned checkpoint is designed to be tolerant with back pressure. AFAIK, this can work with both the default and the new reactive modes.

On Fri, May 14, 2021 at 11:29 PM Alexey Trenikhun <[hidden email]> wrote:

Hello,

Is new reactive mode can operate under back pressure? Old manual rescaling via taking savepoint didn't work with system under back pressure, since it was practically impossible to take savepoint, so wondering is reactive mode expected to be better in this regards ?

Thanks,

Alexey 

[Arvid Heise-4]

Hi Alexey,

Flink supports rescaling from a normal checkpoint if you are not changing your application too much. So if normal checkpointing works, you can just use that for rescaling by using Retained Checkpoints and supply the path on resume at the place where you supplied the savepoint path before. So this is the easiest option to try out since it works on pretty much any Flink version.

Reactive Mode indeed just restores from the latest completed checkpoint. [1] Since it's backed into the system, you don't need to retain checkpoints and supply the path manually. It's rather using the fault tolerance of Flink to rescale on-the-fly. This feature is newly added to Flink 1.13.

The last option is to try out unaligned checkpoints. Since Flink 1.13.0, you can also rescale from unaligned checkpoints. Note that unaligned checkpoints work best if you can use a new source (FileSource or KafkaSource atm).

[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/elastic_scaling/#reactive-mode

On Mon, May 17, 2021 at 4:30 AM Xintong Song <[hidden email]> wrote:

Hi Alexey,

I don't think the new reactive mode makes any changes to the checkpoint/savepoint mechanism, at least not at the moment.

However, you might want to take a look at the unaligned checkpoint [1]. The unaligned checkpoint is designed to be tolerant with back pressure. AFAIK, this can work with both the default and the new reactive modes.

Thank you~

Xintong Song

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/checkpoints/#unaligned-checkpoints

On Fri, May 14, 2021 at 11:29 PM Alexey Trenikhun <[hidden email]> wrote:

Hello,

Is new reactive mode can operate under back pressure? Old manual rescaling via taking savepoint didn't work with system under back pressure, since it was practically impossible to take savepoint, so wondering is reactive mode expected to be better in this regards ?

Thanks,

Alexey 

[Alexey Trenikhun]

Hi Xintong,

Does reactive mode need checkpoint for re-scheduling ?

Thanks,

Alexey

---

From: Xintong Song <[hidden email]>
Sent: Sunday, May 16, 2021 7:30:15 PM
To: Flink User Mail List <[hidden email]>
Subject: Re: reactive mode and back pressure

 

Hi Alexey,

I don't think the new reactive mode makes any changes to the checkpoint/savepoint mechanism, at least not at the moment.

However, you might want to take a look at the unaligned checkpoint [1]. The unaligned checkpoint is designed to be tolerant with back pressure. AFAIK, this can work with both the default and the new reactive modes.

On Fri, May 14, 2021 at 11:29 PM Alexey Trenikhun <[hidden email]> wrote:

Hello,

Is new reactive mode can operate under back pressure? Old manual rescaling via taking savepoint didn't work with system under back pressure, since it was practically impossible to take savepoint, so wondering is reactive mode expected to be better in this regards ?

Thanks,

Alexey 

[Xintong Song]

Yes, it does.

Internally, each re-scheduling is performed as stop-and-resume the job, similar to a failover. Without checkpoints, the job will always restore from the very beginning.

On Mon, May 17, 2021 at 2:54 PM Alexey Trenikhun <[hidden email]> wrote:

Hi Xintong,

Does reactive mode need checkpoint for re-scheduling ?

Thanks,

Alexey

---

From: Xintong Song <[hidden email]>
Sent: Sunday, May 16, 2021 7:30:15 PM
To: Flink User Mail List <[hidden email]>
Subject: Re: reactive mode and back pressure

 

Hi Alexey,

I don't think the new reactive mode makes any changes to the checkpoint/savepoint mechanism, at least not at the moment.

However, you might want to take a look at the unaligned checkpoint [1]. The unaligned checkpoint is designed to be tolerant with back pressure. AFAIK, this can work with both the default and the new reactive modes.

Thank you~

Xintong Song

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/checkpoints/#unaligned-checkpoints

On Fri, May 14, 2021 at 11:29 PM Alexey Trenikhun <[hidden email]> wrote:

Hello,

Is new reactive mode can operate under back pressure? Old manual rescaling via taking savepoint didn't work with system under back pressure, since it was practically impossible to take savepoint, so wondering is reactive mode expected to be better in this regards ?

Thanks,

Alexey