Helm chart for Flink

[Pedro Silva]

Hello,

Forwarding this question from the dev mailing list in case this is a more appropriate list.

Does flink have an official Helm Chart? I haven't been able to find any, the closest most up-to-date one seems to be https://github.com/GoogleCloudPlatform/flink-on-k8s-operator. 

Is this correct or is there a more mature and/or recommeded helm chart to use?

Thank you.

[austin.ce]

Hi Pedro,

There is currently no official Kubernetes Operator for Flink and, by extension, there is no official Helm chart. It would be relatively easy to create a chart for simply deploying standalone Flink resources via the Kubernetes manifests described here[1], though it would leave out the ability to upgrade your Flink application via Helm.

If you need upgrade capabilities (which most people do) and need to use Helm, the Kubernetes Operator approach is the only option for an "all-in-one" experience. In addition to the GCP Operator you mentioned, there's also a Helm chart for Lyft's Operator by lightbend[2] as well as an operator for the Ververica Platform with support for Helm that I've built here[3].

Are you already running Flink on Kubernetes, or just looking to get started easily?

Best,

Austin

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/

[2]: https://github.com/lightbend/flink-operator

[3]: https://github.com/fintechstudios/ververica-platform-k8s-operator/blob/master/docs/guides/deployment.md

On Mon, May 17, 2021 at 11:01 AM Pedro Silva <[hidden email]> wrote:

Hello,

Forwarding this question from the dev mailing list in case this is a more appropriate list.

Does flink have an official Helm Chart? I haven't been able to find any, the closest most up-to-date one seems to be https://github.com/GoogleCloudPlatform/flink-on-k8s-operator. 

Is this correct or is there a more mature and/or recommeded helm chart to use?

Thank you.

[Alexey Trenikhun]

I think it should be possible to use Helm pre-upgrade hook to take savepoint and stop currently running job and then Helm will upgrade image tags. The problem is that if you hit timeout while taking savepoint, it is not clear how to recover from this situation

Alexey

---

From: Austin Cawley-Edwards <[hidden email]>
Sent: Monday, May 17, 2021 1:02 PM
To: Pedro Silva
Cc: user
Subject: Re: Helm chart for Flink

 

Hi Pedro,

There is currently no official Kubernetes Operator for Flink and, by extension, there is no official Helm chart. It would be relatively easy to create a chart for simply deploying standalone Flink resources via the Kubernetes manifests described here[1], though it would leave out the ability to upgrade your Flink application via Helm.

If you need upgrade capabilities (which most people do) and need to use Helm, the Kubernetes Operator approach is the only option for an "all-in-one" experience. In addition to the GCP Operator you mentioned, there's also a Helm chart for Lyft's Operator by lightbend[2] as well as an operator for the Ververica Platform with support for Helm that I've built here[3].

Are you already running Flink on Kubernetes, or just looking to get started easily?

Best,

Austin

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/

[2]: https://github.com/lightbend/flink-operator

[3]: https://github.com/fintechstudios/ververica-platform-k8s-operator/blob/master/docs/guides/deployment.md

On Mon, May 17, 2021 at 11:01 AM Pedro Silva <[hidden email]> wrote:

Hello,

Forwarding this question from the dev mailing list in case this is a more appropriate list.

Does flink have an official Helm Chart? I haven't been able to find any, the closest most up-to-date one seems to be https://github.com/GoogleCloudPlatform/flink-on-k8s-operator. 

Is this correct or is there a more mature and/or recommeded helm chart to use?

Thank you.

[austin.ce]

Hey all,

Yeah, I'd be interested to see the Helm pre-upgrade hook setup, though I'd agree with you, Alexey, that it does not provide enough control to be a stable solution.

[hidden email] I don't know if there are talks for an official operator yet, but Kubernetes support is important to the community and has heavy investment, as seen on the Roadmap[1], so it will only get better! Currently, the recommended way to get started on Kubernetes is with the "Native Kubernetes" resource manager[2], which allows running Flink Applications on Kubernetes from the command line and might work for you.

Hope that helps!

Austin

[1]: https://flink.apache.org/roadmap.html

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes

On Mon, May 17, 2021 at 7:31 PM Alexey Trenikhun <[hidden email]> wrote:

I think it should be possible to use Helm pre-upgrade hook to take savepoint and stop currently running job and then Helm will upgrade image tags. The problem is that if you hit timeout while taking savepoint, it is not clear how to recover from this situation

Alexey

---

From: Austin Cawley-Edwards <[hidden email]>
Sent: Monday, May 17, 2021 1:02 PM
To: Pedro Silva
Cc: user
Subject: Re: Helm chart for Flink

 

Hi Pedro,

There is currently no official Kubernetes Operator for Flink and, by extension, there is no official Helm chart. It would be relatively easy to create a chart for simply deploying standalone Flink resources via the Kubernetes manifests described here[1], though it would leave out the ability to upgrade your Flink application via Helm.

If you need upgrade capabilities (which most people do) and need to use Helm, the Kubernetes Operator approach is the only option for an "all-in-one" experience. In addition to the GCP Operator you mentioned, there's also a Helm chart for Lyft's Operator by lightbend[2] as well as an operator for the Ververica Platform with support for Helm that I've built here[3].

Are you already running Flink on Kubernetes, or just looking to get started easily?

Best,

Austin

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/

[2]: https://github.com/lightbend/flink-operator

[3]: https://github.com/fintechstudios/ververica-platform-k8s-operator/blob/master/docs/guides/deployment.md

On Mon, May 17, 2021 at 11:01 AM Pedro Silva <[hidden email]> wrote:

Hello,

Forwarding this question from the dev mailing list in case this is a more appropriate list.

Does flink have an official Helm Chart? I haven't been able to find any, the closest most up-to-date one seems to be https://github.com/GoogleCloudPlatform/flink-on-k8s-operator. 

Is this correct or is there a more mature and/or recommeded helm chart to use?

Thank you.