Running Beam Pipelines on a Flink Application Mode Cluster

[Jan Bensien]

Hello,

I am currently trying to run my Apache Beam applications using Flink as my backend. Currently i use a session cluster running on Kubernetes. Is it possible to run Beam pipelines using the application mode? I would like to change to application mode, as I currently benchmark my applications with limited execution time and reset my cluster between every execution.

With many thanks,

Jan 

[Yang Wang]

Hi Jan,

If you could run your Apache Beam application with Flink session mode, then it could also

work in application mode. The key difference for application mode is that the job submission happens

in the JobManager pod, not at the Flink client side. If you want to use the standalone application

on k8s[1], then you need to clean up all the resources(e.g. JobManager Job, TaskManager Deployment)

manually after every execution. Maybe you could also have a try on the native Flink application on K8s[2],

all the resources will be cleaned automatically once the job reached the terminal state.

Please note that you need to ensure your user jars already exist in the JobManager pod before

starting the cluster entrypoint. Using the init container or baking the user jars into the docker

image could both work.

[1]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/kubernetes.html#deploy-application-cluster

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/native_kubernetes.html#application-mode

Best,

Yang

Jan Bensien <[hidden email]> 于2021年2月2日周二 上午5:04写道：

Hello,

I am currently trying to run my Apache Beam applications using Flink as my backend. Currently i use a session cluster running on Kubernetes. Is it possible to run Beam pipelines using the application mode? I would like to change to application mode, as I currently benchmark my applications with limited execution time and reset my cluster between every execution.

With many thanks,

Jan