Running Flink Cluster on PaaS

[KristoffSC]

Hi all,
are there any obstacles from running Flink Cluster on PaaS like OpenShift
for example?
Where for example, task manager could be reasigned to different physical
box?
Especially when Flink will be in form of Docker Job Cluster.

Regards,
Krzysztof



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Yang Wang]

Hi KristoffSC,

Currently, Flink could run in containerized environment and i think many users and companies

deploy their production workloads on Kubernetes. For OpenShift, i think it is just like K8s.

You could deploy standalone Flink session/per-job cluster by applying some yamls[1]. Also you

could have a taste on the native integration. The community has not tested on a OpenShift

environment and you could share your feedback after use.

>>  task manager could be reasigned to different physical box?

It does not matter where the taskmanager is assigned. When the network from taskmanager to

jobmanager is normal, then it will register to the jobmanager again.

[1]. https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html

[2]. https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html

KristoffSC <[hidden email]> 于2020年2月21日周五 上午6:43写道：

Hi all,
are there any obstacles from running Flink Cluster on PaaS like OpenShift
for example?
Where for example, task manager could be reasigned to different physical
box?
Especially when Flink will be in form of Docker Job Cluster.

Regards,
Krzysztof



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[KristoffSC]

Thank you Yang Wang,

Regarding [1] and a sentence from that doc.
"This page describes deploying a standalone Flink session"

I always wonder what do you guys mean by "Standalone Flink session" or
"Standalone Cluster" that can be found here [2].

I'm using a Docker with Job Cluster approach, I know that there is also a
Session Cluster docker images. I understand the differences, but I'm not
sure what you are referring to using those to terms from [1] and [2].

Thanks,
Krzysztof

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html#cluster_setup.html
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/jobmanager_high_availability.html





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Yang Wang]

> I always wonder what do you guys mean by "Standalone Flink session" or
"Standalone Cluster" ...

"Standalone Flink session" usually means an empty Flink cluster is started and 

could accept multiple jobs submission from the Flink client or webui. Even all

the jobs finished, the session cluster will still be there until you manually stop it.

"Standalone Flink per-job" means a dedicated Flink cluster is started for only

one the job. It has better isolation. Usually this mode is used in container

environment. And users build in their jars and dependencies in the image.

You could check how to build your custom image and run a per-job cluster here[1].

[1]. https://github.com/apache/flink/blob/release-1.10/flink-container/kubernetes/README.md#deploy-flink-job-cluster

Best,

Yang

KristoffSC <[hidden email]> 于2020年2月21日周五 下午4:03写道：

Thank you Yang Wang,

Regarding [1] and a sentence from that doc.
"This page describes deploying a standalone Flink session"

I always wonder what do you guys mean by "Standalone Flink session" or
"Standalone Cluster" that can be found here [2].

I'm using a Docker with Job Cluster approach, I know that there is also a
Session Cluster docker images. I understand the differences, but I'm not
sure what you are referring to using those to terms from [1] and [2].

Thanks,
Krzysztof

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html#cluster_setup.html
[2]
https://ci.apache.org/projects/flink/flink-docs-stable/ops/jobmanager_high_availability.html





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/