Kubernetes Setup - JM as job vs JM as deployment

[Gil Amsalem]

Hi,

I found that there are 2 different approaches to setup Flink over kubernetes.

1. Deploy job manager as Job.

2. Deploy job manager as Deployment.

What is the recommended way? What are the benefits of each?

Thanks,

[Matthias]

Hi Gil,

I'm not sure whether I understand you correctly. What do you mean by deploying the job manager as "job" or "deployment"? Are you referring to the different deployment modes, Flink offers [1]? These would be independent of Kubernetes. Or do you wonder what the differences are between the Flink on Kubernetes (native) [2] vs Flink on Kubernetes (standalone using YAML files)?

Best,
Matthias

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/#deployment-modes

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html

On Wed, Apr 21, 2021 at 11:19 PM Gil Amsalem <[hidden email]> wrote:

Hi,

I found that there are 2 different approaches to setup Flink over kubernetes.

1. Deploy job manager as Job.

2. Deploy job manager as Deployment.

What is the recommended way? What are the benefits of each?

Thanks,

Gil Amsalem

[Yangze Guo]

Hi, Gil

IIUC, you want to deploy Flink cluster using YAML files yourselves and
want to know whether the JM should be deployed as Job[1] or
Deployment. If that is the case, as Matthias mentioned, Flink provides
two ways to integrate with K8S [2][3], in [3] the JM will be deployed
as a Deployment.

[1] https://kubernetes.io/docs/concepts/workloads/controllers/job/
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html

Best,
Yangze Guo

On Thu, Apr 22, 2021 at 10:46 PM Matthias Pohl <[hidden email]> wrote:

>
> Hi Gil,
> I'm not sure whether I understand you correctly. What do you mean by deploying the job manager as "job" or "deployment"? Are you referring to the different deployment modes, Flink offers [1]? These would be independent of Kubernetes. Or do you wonder what the differences are between the Flink on Kubernetes (native) [2] vs Flink on Kubernetes (standalone using YAML files)?
>
> Best,
> Matthias
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/#deployment-modes
> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/native_kubernetes.html
> [3] https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/kubernetes.html
>
> On Wed, Apr 21, 2021 at 11:19 PM Gil Amsalem <[hidden email]> wrote:
>>
>> Hi,
>>
>> I found that there are 2 different approaches to setup Flink over kubernetes.
>> 1. Deploy job manager as Job.
>> 2. Deploy job manager as Deployment.
>>
>> What is the recommended way? What are the benefits of each?
>>
>> Thanks,
>> Gil Amsalem