Auto/Dynamic scaling in Flink

[Nauroz Khan Nausherwani]

Dear Flink Contributors and users,

I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in stateless or stateful operator.  It would be interesting to know which matrices Flink uses, and when actually Flink triggers auto-scaling.

Any link or reference paper with required information is appreciated.

best regards,

Nauroz

[Tzu-Li (Gordon) Tai]

Hi,

Flink does not support auto-scaling, yet. Rescaling operations currently are always manual, i.e take a savepoint of the Flink job, and when restoring from the savepoint, define a new parallelism for the job.
As for the metrics to be used for auto-scaling, I can imagine that it would be possible to base this on top of metrics such as TM throughput, backpressure, etc. AFAIK, there also currently isn't any official design or discussion on going for this.

Cheers,
Gordon

On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <[hidden email]> wrote:

Dear Flink Contributors and users,

I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in stateless or stateful operator.  It would be interesting to know which matrices Flink uses, and when actually Flink triggers auto-scaling.

Any link or reference paper with required information is appreciated.

best regards,

Nauroz

[tison]

Hi,

Yet Flink does not support auto-scaling. However, there is an umbrella JIRA issue[1]

to cover the discussion about it. And I think the design doc(draft) attached[2] could help.

[2] https://docs.google.com/document/d/1XKDXnrp8w45k2jIJNHxpNP2Zmr6FGru3H0dTHwYWJyE/edit

Tzu-Li (Gordon) Tai <[hidden email]> 于2018年11月13日周二 下午8:05写道：

Hi,

Flink does not support auto-scaling, yet. Rescaling operations currently are always manual, i.e take a savepoint of the Flink job, and when restoring from the savepoint, define a new parallelism for the job.
As for the metrics to be used for auto-scaling, I can imagine that it would be possible to base this on top of metrics such as TM throughput, backpressure, etc. AFAIK, there also currently isn't any official design or discussion on going for this.

Cheers,
Gordon

On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <[hidden email]> wrote:

Dear Flink Contributors and users,

I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in stateless or stateful operator.  It would be interesting to know which matrices Flink uses, and when actually Flink triggers auto-scaling.

Any link or reference paper with required information is appreciated.

best regards,

Nauroz

[罗齐]

Hi Nauroz,

If you’re using Flink 1.5 on Yarn, it supports dynamic task manager allocation by default [1]. After skimming the code, it seems to me that in general if requested parallelism is larger than available task slots, new task managers will be requested via ResourceManager (please correct me if any misunderstanding).

We’re also looking into auto scaling issue in Flink, as it’s crucial in our use cases (e.g. in batch mode, we will do input with few TMs but output with much more TMs). Please kindly let me know if any further information are available.

[1] https://stackoverflow.com/questions/38054015/can-yarn-dynamically-allocate-resources-to-flink/38077626#38077626

On Nov 13, 2018, at 8:10 PM, Tzu-Li Chen <[hidden email]> wrote:

Hi,

Yet Flink does not support auto-scaling. However, there is an umbrella JIRA issue[1]

to cover the discussion about it. And I think the design doc(draft) attached[2] could help.

[2] https://docs.google.com/document/d/1XKDXnrp8w45k2jIJNHxpNP2Zmr6FGru3H0dTHwYWJyE/edit

Tzu-Li (Gordon) Tai <[hidden email]> 于2018年11月13日周二 下午8:05写道：

Hi,

Flink does not support auto-scaling, yet. Rescaling operations currently are always manual, i.e take a savepoint of the Flink job, and when restoring from the savepoint, define a new parallelism for the job.
As for the metrics to be used for auto-scaling, I can imagine that it would be possible to base this on top of metrics such as TM throughput, backpressure, etc. AFAIK, there also currently isn't any official design or discussion on going for this.

Cheers,
Gordon

On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <[hidden email]> wrote:

Dear Flink Contributors and users,

I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in stateless or stateful operator.  It would be interesting to know which matrices Flink uses, and when actually Flink triggers auto-scaling.

Any link or reference paper with required information is appreciated.

best regards,

Nauroz