Native Kubernetes deployment with GitOps

[Alex Adriaanse]

I'm trying to setup a Flink 1.12 deployment on a Kubernetes cluster using custom Docker images (since the official ones aren't out yet). Since the documentation states that "We generally recommend new users to deploy Flink on Kubernetes using native Kubernetes deployments", I'm trying out the native Kubernetes integration.

At the same time, I'm also trying to move to GitOps where we use a tool like Argo CD to keep Kubernetes resources in sync with declarations stored in a Git repository. From what I can tell there's no way to do GitOps with native Kubernetes deployments, since the Kubernetes resources it produces are generated dynamically and are managed via CLI rather than being suitable for storage in a Git repository. Is that correct?

Would it be better in my case to just use the standalone Kubernetes deployment option and have our CD tool deploy the YAML files listed in Flink's documentation via Git? Or should I use the native Kubernetes integration to create an initial set of resources in our cluster, export those to a Git repository, and then use a GitOps workflow to modify those resources going forward rather than using the Flink CLI? In other words, I'd only use the Flink CLI to bootstrap the Git repository.

Besides ease of setup, are there any other benefits to the native Kubernetes integration compared to standalone deployments?

Thanks!

Alex

[Israel Ekpo]

If you are just getting started, running it on Kubernetes could simplify that logistics and resources needed for getting started.

It also allows you to possibly reuse infrastructure that you may already be using for other projects and purposes.

If you are just getting started and just learning, the setup and teardown time is much faster with Kubernetes (in my experience) when compared to setting it as Standalone on VMs 

Network connectivity to other components within my ecosystem is also easier when with K8s.

Other options are available here for your reference

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/#overview-and-reference-architecture

If you are just starting out, I would recommend deploying your containers on the K8S cluster via YAML manifests first and then convert to Helm charts (once the vanilla YAML is stable) so that you can bundle the various components as a single deployment.

On Thu, Jan 7, 2021 at 4:57 PM Alex Adriaanse <[hidden email]> wrote:

I'm trying to setup a Flink 1.12 deployment on a Kubernetes cluster using custom Docker images (since the official ones aren't out yet). Since the documentation states that "We generally recommend new users to deploy Flink on Kubernetes using native Kubernetes deployments", I'm trying out the native Kubernetes integration.

At the same time, I'm also trying to move to GitOps where we use a tool like Argo CD to keep Kubernetes resources in sync with declarations stored in a Git repository. From what I can tell there's no way to do GitOps with native Kubernetes deployments, since the Kubernetes resources it produces are generated dynamically and are managed via CLI rather than being suitable for storage in a Git repository. Is that correct?

Would it be better in my case to just use the standalone Kubernetes deployment option and have our CD tool deploy the YAML files listed in Flink's documentation via Git? Or should I use the native Kubernetes integration to create an initial set of resources in our cluster, export those to a Git repository, and then use a GitOps workflow to modify those resources going forward rather than using the Flink CLI? In other words, I'd only use the Flink CLI to bootstrap the Git repository.

Besides ease of setup, are there any other benefits to the native Kubernetes integration compared to standalone deployments?

Thanks!

Alex

[Alex Adriaanse]

Thanks for the reply!

Just to clarify, when I talked about standalone deployments I was referring to standalone Kubernetes deployments. We currently have no interest in running Flink outside of K8s. I was mostly just curious about the differences in the native integration vs. standalone deployment options on Kubernetes.

As a side note, we're planning on using Flink in Application Mode.

Alex

On Jan 7, 2021, at 4:17 PM, Israel Ekpo <[hidden email]> wrote:

If you are just getting started, running it on Kubernetes could simplify that logistics and resources needed for getting started.

It also allows you to possibly reuse infrastructure that you may already be using for other projects and purposes.

If you are just getting started and just learning, the setup and teardown time is much faster with Kubernetes (in my experience) when compared to setting it as Standalone on VMs 

Network connectivity to other components within my ecosystem is also easier when with K8s.

Other options are available here for your reference

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/#overview-and-reference-architecture

If you are just starting out, I would recommend deploying your containers on the K8S cluster via YAML manifests first and then convert to Helm charts (once the vanilla YAML is stable) so that you can bundle the various components as a single deployment.

On Thu, Jan 7, 2021 at 4:57 PM Alex Adriaanse <[hidden email]> wrote:

I'm trying to setup a Flink 1.12 deployment on a Kubernetes cluster using custom Docker images (since the official ones aren't out yet). Since the documentation states that "We generally recommend new users to deploy Flink on Kubernetes using native Kubernetes deployments", I'm trying out the native Kubernetes integration.

At the same time, I'm also trying to move to GitOps where we use a tool like Argo CD to keep Kubernetes resources in sync with declarations stored in a Git repository. From what I can tell there's no way to do GitOps with native Kubernetes deployments, since the Kubernetes resources it produces are generated dynamically and are managed via CLI rather than being suitable for storage in a Git repository. Is that correct?

Would it be better in my case to just use the standalone Kubernetes deployment option and have our CD tool deploy the YAML files listed in Flink's documentation via Git? Or should I use the native Kubernetes integration to create an initial set of resources in our cluster, export those to a Git repository, and then use a GitOps workflow to modify those resources going forward rather than using the Flink CLI? In other words, I'd only use the Flink CLI to bootstrap the Git repository.

Besides ease of setup, are there any other benefits to the native Kubernetes integration compared to standalone deployments?

Thanks!

Alex