Operators resource requirements on K8s Flink session cluster

[Michaël Melchiore]

Hello,

I plan to run topologies on a Flink session cluster on Kubernetes.

In my topologies, operators will have varying resource requirements in term of CPU and RAM. 

How can I make these informations available from Flink to Kubernetes so the latter takes it into account to optimize its deployment ?

I am trying to achieve something similar to Apache Storm/Trident Resource Aware Scheduler.

Kind regards,

Michaël

[Yang Wang]

Hi Michaël,

Glad to hear that you are going to run Flink workload on Kubernetes. AFAIK, we have two

deployment ways.

1. Running Flink standalone session/per-job cluster on K8s. You need to calculate how many

taskmanagers you need and the <memory, cpu> per taskmanager. All the taskmanager

will be started by a K8s deployment. You could find more information here[1]. In this mode,

you could be `kubectl scale` to change the replicas of taskmanager if the resources are not

enough for your job.

2. Natively running Flink session/per-job on K8s. The session mode has been support in 

master branch and will be released in 1.10. The per-job mode is in discussion. No matter

session or per-job, the taskmanager will be allocated dynamically on demand. You could

use a simple command to start a Flink cluster on K8s. More information could be found

here[2].

Best,

Yang

[1]. https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html

[2]. https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing

Michaël Melchiore <[hidden email]> 于2019年12月19日周四 上午1:11写道：

Hello,

I plan to run topologies on a Flink session cluster on Kubernetes.

In my topologies, operators will have varying resource requirements in term of CPU and RAM. 

How can I make these informations available from Flink to Kubernetes so the latter takes it into account to optimize its deployment ?

I am trying to achieve something similar to Apache Storm/Trident Resource Aware Scheduler.

Kind regards,

Michaël

[Michaël Melchiore]

Hi Yang,

Thanks for your quick reply.

The Flink K8s documentation distinguishes between standalone and session deployment mode. I think I will use the latter.

Since my previous mail, I found FLIP-53 which is precisely the topic of my original question. 

So, great progress has been already made to cover my needs. Unfortunately, I am use DataStreams API which are currently not covered by the initial implementation. I have asked on the dev mailing list if I could help bridging this gap.

Regards,

Michaël

Le jeu. 19 déc. 2019 à 04:58, Yang Wang <[hidden email]> a écrit :

Hi Michaël,

Glad to hear that you are going to run Flink workload on Kubernetes. AFAIK, we have two

deployment ways.

1. Running Flink standalone session/per-job cluster on K8s. You need to calculate how many

taskmanagers you need and the <memory, cpu> per taskmanager. All the taskmanager

will be started by a K8s deployment. You could find more information here[1]. In this mode,

you could be `kubectl scale` to change the replicas of taskmanager if the resources are not

enough for your job.

2. Natively running Flink session/per-job on K8s. The session mode has been support in 

master branch and will be released in 1.10. The per-job mode is in discussion. No matter

session or per-job, the taskmanager will be allocated dynamically on demand. You could

use a simple command to start a Flink cluster on K8s. More information could be found

here[2].

Best,

Yang

[1]. https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html

[2]. https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing

Michaël Melchiore <[hidden email]> 于2019年12月19日周四 上午1:11写道：

Hello,

I plan to run topologies on a Flink session cluster on Kubernetes.

In my topologies, operators will have varying resource requirements in term of CPU and RAM. 

How can I make these informations available from Flink to Kubernetes so the latter takes it into account to optimize its deployment ?

I am trying to achieve something similar to Apache Storm/Trident Resource Aware Scheduler.

Kind regards,

Michaël