Dynamic scaling in flink

[Abhishek Agarwal]

Is it possible to upscale or downscale a flink application without re-deploying (similar to rebalancing in storm)?

--

[Stephan Ewen]

Hi!

There is a lot of work in progress on that feature, and it looks like you can expect the next version to have some upscale/downscale feature that maintains exactly-once semantics.

Stephan

On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal <[hidden email]> wrote:

Is it possible to upscale or downscale a flink application without re-deploying (similar to rebalancing in storm)?

--

[Abhishek Agarwal]

Thanks Stephan. I understand guaranteeing exactly once semantics with the dynamic scaling is tough. If I were to let go of the exactly once requirement, is it not possible in current version? It would be really great if you can point me to the JIRA tracking this work. 

On Mon, Aug 29, 2016 at 2:30 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

There is a lot of work in progress on that feature, and it looks like you can expect the next version to have some upscale/downscale feature that maintains exactly-once semantics.

Stephan

On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal <[hidden email]> wrote:

Is it possible to upscale or downscale a flink application without re-deploying (similar to rebalancing in storm)?

--

--

[rmetzger0]

Hi,

this JIRA is a good starting point: https://issues.apache.org/jira/browse/FLINK-3755

If you don't care about processing guarantees and you are using a stateless streaming job, you can implement a simple Kafka consumer that uses Kafka's consumer group mechanism. I recently implemented such a Kafka consumer if you want to use it as a base for your work: https://github.com/rmetzger/scratch/blob/kafka-group-consumer/src/main/java/com/dataartisans/FlinkKafkaGroupConsumer.java

Regards,

Robert

On Mon, Aug 29, 2016 at 11:54 AM, Abhishek Agarwal <[hidden email]> wrote:

Thanks Stephan. I understand guaranteeing exactly once semantics with the dynamic scaling is tough. If I were to let go of the exactly once requirement, is it not possible in current version? It would be really great if you can point me to the JIRA tracking this work. 

On Mon, Aug 29, 2016 at 2:30 PM, Stephan Ewen <[hidden email]> wrote:

Hi!

There is a lot of work in progress on that feature, and it looks like you can expect the next version to have some upscale/downscale feature that maintains exactly-once semantics.

Stephan

On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal <[hidden email]> wrote:

Is it possible to upscale or downscale a flink application without re-deploying (similar to rebalancing in storm)?

--

--