Auto Scaling in Flink

[Akash Goel]

Hi,

Does Flunk support auto scaling. I read that it is supported using pravega? Is it incorporated in any version.

Thanks,

Akash Goel

[vino yang]

Hi Akash,

You can use Pravega connector to integrate with Flink, the source code is here[1].

In short, relying on its rescalable state feature[2] flink supports scalable streaming jobs.

Currently, the mainstream solution about auto-scaling is Flink + K8S, I can share some resources with you[3].

Best,

Vino

[1]: https://github.com/pravega/flink-connectors

[2]: https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html

[3]: https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam

Akash Goel <[hidden email]> 于2019年11月29日周五 上午9:52写道：

Hi,

Does Flunk support auto scaling. I read that it is supported using pravega? Is it incorporated in any version.

Thanks,

Akash Goel

[Caizhi Weng]

Hi Akash,

Flink doesn't support auto scaling in core currently, it may be supported in the future, when the new scheduling architecture is implemented https://issues.apache.org/jira/browse/FLINK-10407 .

You can do it externally by cancel the job with a savepoint, update the parallelism, and restart the job, according to the rate of data. like what pravega suggests in the doc: http://pravega.io/docs/latest/key-features/#auto-scaling.

vino yang <[hidden email]> 于2019年11月29日周五 上午11:12写道：

Hi Akash,

You can use Pravega connector to integrate with Flink, the source code is here[1].

In short, relying on its rescalable state feature[2] flink supports scalable streaming jobs.

Currently, the mainstream solution about auto-scaling is Flink + K8S, I can share some resources with you[3].

Best,

Vino

[1]: https://github.com/pravega/flink-connectors

[2]: https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html

[3]: https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam

Akash Goel <[hidden email]> 于2019年11月29日周五 上午9:52写道：

Hi,

Does Flunk support auto scaling. I read that it is supported using pravega? Is it incorporated in any version.

Thanks,

Akash Goel

[Akash Goel]

Hi,

If my application is already running on Kafka, then do I need to replace with Pravega or can Pravega read directly from Kafka?

I have also reached out to to Pravega Community but just checking here.

Thanks,

Akash Goel

On Fri, Nov 29, 2019 at 11:14 AM Caizhi Weng <[hidden email]> wrote:

Hi Akash,

Flink doesn't support auto scaling in core currently, it may be supported in the future, when the new scheduling architecture is implemented https://issues.apache.org/jira/browse/FLINK-10407 .

You can do it externally by cancel the job with a savepoint, update the parallelism, and restart the job, according to the rate of data. like what pravega suggests in the doc: http://pravega.io/docs/latest/key-features/#auto-scaling.

vino yang <[hidden email]> 于2019年11月29日周五 上午11:12写道：

Hi Akash,

You can use Pravega connector to integrate with Flink, the source code is here[1].

In short, relying on its rescalable state feature[2] flink supports scalable streaming jobs.

Currently, the mainstream solution about auto-scaling is Flink + K8S, I can share some resources with you[3].

Best,

Vino

[1]: https://github.com/pravega/flink-connectors

[2]: https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html

[3]: https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam

Akash Goel <[hidden email]> 于2019年11月29日周五 上午9:52写道：

Hi,

Does Flunk support auto scaling. I read that it is supported using pravega? Is it incorporated in any version.

Thanks,

Akash Goel

[vino yang]

Hi Akash,

The key difference between Pravega and Kafka is: Kafka is a messaging system, while Pravega is a streaming system.[1] The official documentation also statements their difference in their faq page.[2]

[1]: https://siliconangle.com/2017/04/17/dell-emc-takes-on-streaming-storage-with-open-source-solution-pravega-ffsf17/

[2]: http://www.pravega.io/faq.html

Best,

Vino

Akash Goel <[hidden email]> 于2019年12月4日周三 下午12:00写道：

Hi,

If my application is already running on Kafka, then do I need to replace with Pravega or can Pravega read directly from Kafka?

I have also reached out to to Pravega Community but just checking here.

Thanks,

Akash Goel

On Fri, Nov 29, 2019 at 11:14 AM Caizhi Weng <[hidden email]> wrote:

Hi Akash,

Flink doesn't support auto scaling in core currently, it may be supported in the future, when the new scheduling architecture is implemented https://issues.apache.org/jira/browse/FLINK-10407 .

You can do it externally by cancel the job with a savepoint, update the parallelism, and restart the job, according to the rate of data. like what pravega suggests in the doc: http://pravega.io/docs/latest/key-features/#auto-scaling.

vino yang <[hidden email]> 于2019年11月29日周五 上午11:12写道：

Hi Akash,

You can use Pravega connector to integrate with Flink, the source code is here[1].

In short, relying on its rescalable state feature[2] flink supports scalable streaming jobs.

Currently, the mainstream solution about auto-scaling is Flink + K8S, I can share some resources with you[3].

Best,

Vino

[1]: https://github.com/pravega/flink-connectors

[2]: https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html

[3]: https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2019-scaling-a-realtime-streaming-warehouse-with-apache-flink-parquet-and-kubernetes-aditi-verma-ramesh-shanmugam

Akash Goel <[hidden email]> 于2019年11月29日周五 上午9:52写道：

Hi,

Does Flunk support auto scaling. I read that it is supported using pravega? Is it incorporated in any version.

Thanks,

Akash Goel