Re: Flink on Kubernetes Vs Flink Natively on Kubernetes

Posted by Xintong Song on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-on-Kubernetes-Vs-Flink-Natively-on-Kubernetes-tp33666p33673.html

Forgot to mention that "running Flink natively on Kubernetes" is newly introduced and is only available for Flink 1.10 and above.

On Mon, Mar 16, 2020 at 5:40 PM Xintong Song <[hidden email]> wrote:

Hi Pankaj,

"Running Flink on Kubernetes" refers to the old way that basically deploys a Flink standalone cluster on Kubernetes. We leverage scripts to run Flink Master and TaskManager processes inside Kubernetes container. In this way, Flink is not ware of whether it's running in containers or directly on physical machines, and will not interact with the Kubernetes Master. Flink Master reactively accept all registered TaskManagers, whose number is decided by the Kubernetes replica. 

"Running Flink natively on Kubernetes" refers deploy Flink as a Kubernetes Job. Flink Master will interact with Kubernetes Master, and actively requests for pods/containers, like on Yarn/Mesos.

Thank you~

Xintong Song

On Mon, Mar 16, 2020 at 4:03 PM Pankaj Chand <[hidden email]> wrote:

Hi all,

I want to run Flink, Spark and other processing engines on a single Kubernetes cluster.

From the Flink documentation, I did not understand the difference between:

(1) Running Flink on Kubernetes, Versus (2) Running Flink natively on Kubernetes.

Could someone please explain the difference between the two, and when would you use which option?

Thank you,

Pankaj