Re: Max-parellelism limitation

Posted by Olivier Nouguier on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Max-parellelism-limitation-tp43059p43081.html

Hi, thank you all for your reply, by limitation I meant the impossibility to resume a job when scaling up because of this max-parallelism.

To be more precise, in our deployment, the operator (max) parallelism is computed from the number of available slots ( ~~ task-manager * core ), this approach is probably too naive, at least with keyed state.

On Tue, Apr 20, 2021 at 10:46 AM Chesnay Schepler <[hidden email]> wrote:

@Olivier Could you clarify which limitation you are referring to?

On 4/20/2021 5:23 AM, Guowei Ma wrote:

Hi, Olivier

Yes. The introduction of this concept is to solve the problem of rescaling the keystate.

Best,

Guowei

On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <[hidden email]> wrote:

Hi,

  May I have the confirmation that the max-parallelism limitation only occurs when keyed states are used ?

--

Olivier Nouguier SSE e | [hidden email] m | 0651383971 Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 Montpellier, France

The information in this email is confidential and intended only for the addressee(s) named above. If you are not the intended recipient any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. Teads does not warrant that any attachment(s) are free from viruses or other defects and accept no liability for any losses resulting from infected email transmission. Please note that any views expressed in this email may be those of the originator and do not necessarily reflect those of the organization.

--