Deploying native Kubernetes via yaml files?

[Niels Basjes]

Hi,

As clearly documented here https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html the current way of deploying Flink natively on Kubernetes is by running the ./bin/kubernetes-session.sh script that runs some Java code that does "magic" to deploy in on the cluster.

This works.

I was wondering: Is it with the current code base already possible to craft a set of Yaml files (perhaps even with a special Docker image) so that I can deploy it using the 'normal' Kubernetes way of doing a kubectl apply -f foo.yaml ?

--

[Ufuk Celebi]

Hey Niels,

you can check out the README with example configuration files here: https://github.com/apache/flink/tree/master/flink-container/kubernetes

Is that what you were looking for?

Best,

Ufuk

On Tue, Mar 24, 2020 at 2:42 PM Niels Basjes <[hidden email]> wrote:

Hi,

As clearly documented here https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html the current way of deploying Flink natively on Kubernetes is by running the ./bin/kubernetes-session.sh script that runs some Java code that does "magic" to deploy in on the cluster.

This works.

I was wondering: Is it with the current code base already possible to craft a set of Yaml files (perhaps even with a special Docker image) so that I can deploy it using the 'normal' Kubernetes way of doing a kubectl apply -f foo.yaml ?

--

Best regards / Met vriendelijke groeten,

Niels Basjes

[Ufuk Celebi]

PS: See also https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html#flink-job-cluster-on-kubernetes

On Tue, Mar 24, 2020 at 2:49 PM Ufuk Celebi <[hidden email]> wrote:

Hey Niels,

you can check out the README with example configuration files here: https://github.com/apache/flink/tree/master/flink-container/kubernetes

Is that what you were looking for?

Best,

Ufuk

On Tue, Mar 24, 2020 at 2:42 PM Niels Basjes <[hidden email]> wrote:

Hi,

As clearly documented here https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html the current way of deploying Flink natively on Kubernetes is by running the ./bin/kubernetes-session.sh script that runs some Java code that does "magic" to deploy in on the cluster.

This works.

I was wondering: Is it with the current code base already possible to craft a set of Yaml files (perhaps even with a special Docker image) so that I can deploy it using the 'normal' Kubernetes way of doing a kubectl apply -f foo.yaml ?

--

Best regards / Met vriendelijke groeten,

Niels Basjes

[Yang Wang]

Hi Niels,

Currently, the native integration Flink cluster could not be created via yaml file. The reason

why we introduce the native integration is for the users who are not familiar with K8s and

kubectl. So we want to make it easier for our Flink users to deploy Flink cluster on K8s.

However, i think some other users, especially K8s experts, they really like the yaml way to

create Flink cluster. I will create a ticket for this feature. If you want to do this manually now,

i think you need to at least do the following changes. 

1. Update the `flink-console.sh` and add a new script(mostly like jobmanager.sh) to set the

entrypoint to "org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint".

2. Create the configmap, service like the standalone on K8s[1].

3. Create a jobmanager.yaml with service account(enough permission to create pod in K8s

cluster) correctly set

[1]. https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html#appendix

Best,

Yang

Ufuk Celebi <[hidden email]> 于2020年3月24日周二 下午9:51写道：

PS: See also https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html#flink-job-cluster-on-kubernetes

On Tue, Mar 24, 2020 at 2:49 PM Ufuk Celebi <[hidden email]> wrote:

Hey Niels,

you can check out the README with example configuration files here: https://github.com/apache/flink/tree/master/flink-container/kubernetes

Is that what you were looking for?

Best,

Ufuk

On Tue, Mar 24, 2020 at 2:42 PM Niels Basjes <[hidden email]> wrote:

Hi,

As clearly documented here https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html the current way of deploying Flink natively on Kubernetes is by running the ./bin/kubernetes-session.sh script that runs some Java code that does "magic" to deploy in on the cluster.

This works.

I was wondering: Is it with the current code base already possible to craft a set of Yaml files (perhaps even with a special Docker image) so that I can deploy it using the 'normal' Kubernetes way of doing a kubectl apply -f foo.yaml ?

--

Best regards / Met vriendelijke groeten,

Niels Basjes

[Yang Wang]

Hi Niels,

I have created a ticket[1] to track the yaml file submission for native K8s integration.

Feel free to share your significative thoughts about this way.

[1]. https://issues.apache.org/jira/browse/FLINK-16760

Best,

Yang

Niels Basjes <[hidden email]> 于2020年3月24日周二 下午11:10写道：

Thanks.

I'll have a look at this.

Can you post the ticket number for this feature (after you created it) please?

Niels

On Tue, Mar 24, 2020 at 3:54 PM Yang Wang <[hidden email]> wrote:

Hi Niels,

Currently, the native integration Flink cluster could not be created via yaml file. The reason

why we introduce the native integration is for the users who are not familiar with K8s and

kubectl. So we want to make it easier for our Flink users to deploy Flink cluster on K8s.

However, i think some other users, especially K8s experts, they really like the yaml way to

create Flink cluster. I will create a ticket for this feature. If you want to do this manually now,

i think you need to at least do the following changes. 

1. Update the `flink-console.sh` and add a new script(mostly like jobmanager.sh) to set the

entrypoint to "org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint".

2. Create the configmap, service like the standalone on K8s[1].

3. Create a jobmanager.yaml with service account(enough permission to create pod in K8s

cluster) correctly set

[1]. https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html#appendix

Best,

Yang

Ufuk Celebi <[hidden email]> 于2020年3月24日周二 下午9:51写道：

PS: See also https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html#flink-job-cluster-on-kubernetes

On Tue, Mar 24, 2020 at 2:49 PM Ufuk Celebi <[hidden email]> wrote:

Hey Niels,

you can check out the README with example configuration files here: https://github.com/apache/flink/tree/master/flink-container/kubernetes

Is that what you were looking for?

Best,

Ufuk

On Tue, Mar 24, 2020 at 2:42 PM Niels Basjes <[hidden email]> wrote:

Hi,

As clearly documented here https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html the current way of deploying Flink natively on Kubernetes is by running the ./bin/kubernetes-session.sh script that runs some Java code that does "magic" to deploy in on the cluster.

This works.

I was wondering: Is it with the current code base already possible to craft a set of Yaml files (perhaps even with a special Docker image) so that I can deploy it using the 'normal' Kubernetes way of doing a kubectl apply -f foo.yaml ?

--

Best regards / Met vriendelijke groeten,

Niels Basjes

--

Best regards / Met vriendelijke groeten,

Niels Basjes