Automatic deployment of new version of streaming stateful job

[Maxim Parkachov]

Hi,

I'm trying to bring my first stateful streaming Flink job to production and have trouble understanding how to integrate it with CI/CD pipeline. I can cancel the job with savepoint, but in order to start new version of application I need to specify savepoint path manually ? 

So, basically my question, what is best practice of automatically restarting or deploying new version of stateful streaming application ? Every tip is greatly appreciated.

Thanks,

Maxim.

[Marc Rooding]

Hi Maxim

You could write a script yourself which triggers the cancel with savepoint and then starts a new version using the savepoint that was created during the cancel. 

However, I’ve built a tool that allows you to perform these steps more easily: https://github.com/ing-bank/flink-deployer. The deployer will allow you to deploy or upgrade your jobs. All you need to do is integrate it into your CI/CD.

Kind regards

Marc

On 16 Jul 2019, 02:46 +0200, Maxim Parkachov <[hidden email]>, wrote:

Hi,

I'm trying to bring my first stateful streaming Flink job to production and have trouble understanding how to integrate it with CI/CD pipeline. I can cancel the job with savepoint, but in order to start new version of application I need to specify savepoint path manually ? 

So, basically my question, what is best practice of automatically restarting or deploying new version of stateful streaming application ? Every tip is greatly appreciated.

Thanks,

Maxim.

[Maxim Parkachov]

Hi Marc,

thanks a lot for the tool. Unfortunately, I could not direcly use it, but I will take couple of ideas and will implement my own script.

Nevertherless, I'm really surprised that such functionality doesn't exist out of the box.

Regards,

Maxim.

On Tue, Jul 16, 2019 at 9:22 AM Marc Rooding <[hidden email]> wrote:

Hi Maxim

You could write a script yourself which triggers the cancel with savepoint and then starts a new version using the savepoint that was created during the cancel. 

However, I’ve built a tool that allows you to perform these steps more easily: https://github.com/ing-bank/flink-deployer. The deployer will allow you to deploy or upgrade your jobs. All you need to do is integrate it into your CI/CD.

Kind regards

Marc

On 16 Jul 2019, 02:46 +0200, Maxim Parkachov <[hidden email]>, wrote:

Hi,

I'm trying to bring my first stateful streaming Flink job to production and have trouble understanding how to integrate it with CI/CD pipeline. I can cancel the job with savepoint, but in order to start new version of application I need to specify savepoint path manually ? 

So, basically my question, what is best practice of automatically restarting or deploying new version of stateful streaming application ? Every tip is greatly appreciated.

Thanks,

Maxim.