manual scaling with savepoint

[gallenvara]

Hi, everyone. Now, Flink can't do with auto-scaling and  we can realize this by restart the savepoint with different parallelism. I wonder how flink handle with state managerment. For example, for parallelism=3 to 4, how the state of 3 deal with new parallelism? Can you explain the internal to me ? Thanks a lot.

[Fabian Hueske-2]

Hi,

Flink supports two types of state:

1) Key-partitioned state

2) Non-partitioned operator state (Checkpointed interface)

Key-partitioned state is internally organized by key and can be "simply" rehashed. The actual implementation is more involved to make this efficient. This document contains details [1].

In general, non-partitioned state can not be split. However, there are special types of state which is not organized by key but which is splittable such as a list state which can be split into individual elements and redistributed.

This is described in this design document [2].

Hope this helps,

Fabian

[1] https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit#heading=h.2suhu1fjxmp4
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-8%3A+Rescalable+Non-Partitioned+State

2017-01-11 8:34 GMT+01:00 gallenvara <[hidden email]>:

Hi, everyone. Now, Flink can't do with auto-scaling and  we can realize this
by restart the savepoint with different parallelism. I wonder how flink
handle with state managerment. For example, for parallelism=3 to 4, how the
state of 3 deal with new parallelism? Can you explain the internal to me ?
Thanks a lot.





--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/manual-scaling-with-savepoint-tp10974.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[gallenvara]

Thanks for your detail explanation.