Max-parellelism limitation

[Olivier Nouguier]

Hi,

  May I have the confirmation that the max-parallelism limitation only occurs when keyed states are used ?

--

[Guowei Ma]

Hi, Olivier

Yes. The introduction of this concept is to solve the problem of rescaling the keystate.

On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <[hidden email]> wrote:

Hi,

  May I have the confirmation that the max-parallelism limitation only occurs when keyed states are used ?

--

Olivier Nouguier

SSE

e | [hidden email] m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.

[Chesnay Schepler]

@Olivier Could you clarify which limitation you are referring to?

On 4/20/2021 5:23 AM, Guowei Ma wrote:

Hi, Olivier

Yes. The introduction of this concept is to solve the problem of rescaling the keystate.

On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <[hidden email]> wrote:

Hi,

  May I have the confirmation that the max-parallelism limitation only occurs when keyed states are used ?

--

Olivier Nouguier

SSE

e | [hidden email] m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.

[Olivier Nouguier]

Hi, thank you all for your reply, by limitation I meant the impossibility to resume a job when scaling up because of this max-parallelism.

To be more precise, in our deployment, the operator (max) parallelism is computed from the number of available slots ( ~~ task-manager * core ), this approach is probably too naive, at least with keyed state.

On Tue, Apr 20, 2021 at 10:46 AM Chesnay Schepler <[hidden email]> wrote:

@Olivier Could you clarify which limitation you are referring to?

On 4/20/2021 5:23 AM, Guowei Ma wrote:

Hi, Olivier

Yes. The introduction of this concept is to solve the problem of rescaling the keystate.

Best,

Guowei

On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <[hidden email]> wrote:

Hi,

  May I have the confirmation that the max-parallelism limitation only occurs when keyed states are used ?

--

Olivier Nouguier

SSE

e | [hidden email] m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.

--

[austin.ce]

Hi Olivier,

Someone will correct me if I'm wrong, but I believe the max-parallelism limitation, where you cannot scale up past the previously defined max-parallelism, applies to all stateful jobs no matter which type of state you are using.

If you haven't seen it already, I think the Production Readiness checklist [1] offers a good explanation for this behavior/ best practices for setting it.

Best,

Austin

[1]: https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/production_ready/#set-an-explicit-max-parallelism

On Tue, Apr 20, 2021 at 9:33 AM Olivier Nouguier <[hidden email]> wrote:

Hi, thank you all for your reply, by limitation I meant the impossibility to resume a job when scaling up because of this max-parallelism.

To be more precise, in our deployment, the operator (max) parallelism is computed from the number of available slots ( ~~ task-manager * core ), this approach is probably too naive, at least with keyed state.

On Tue, Apr 20, 2021 at 10:46 AM Chesnay Schepler <[hidden email]> wrote:

@Olivier Could you clarify which limitation you are referring to?

On 4/20/2021 5:23 AM, Guowei Ma wrote:

Hi, Olivier

Yes. The introduction of this concept is to solve the problem of rescaling the keystate.

Best,

Guowei

On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <[hidden email]> wrote:

Hi,

  May I have the confirmation that the max-parallelism limitation only occurs when keyed states are used ?

--

Olivier Nouguier

SSE

e | [hidden email] m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.

--

Olivier Nouguier

SSE

e | [hidden email] m | 0651383971

Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.