Re: Job Cluster on Kubernetes with PyFlink
Posted by
Shuiqiang Chen on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Job-Cluster-on-Kubernetes-with-PyFlink-tp37001p37006.html
Hi Wojciech,
After double checking, there should be a way to run PyFlink jobs on kubernetes in the job cluster. You can have a try:
1. The custom image has a corresponding pyflink installed. (it seems that you have already done this)
2. If you use third-party python dependencies in the Python UDF, please make sure that the Python dependencies should also be pip installed
3. Putting the flink-python_{your_scala_version}-{your_flink_version}.jar into the /opt/flink/usrlib directory when building the custom docker image.
4. Setting the value of option "--job-classname" to be "org.apache.flink.client.python.PythonDriver".
5. Adding '-pym {the_entry_module_of_your_pyflink_job}' to [job arguments].
Best,
Shuiqiang
Hi Wojciech,
Currently, we are not able to deploy a job cluster for PyFlink jobs on kubernetes, but it will be supported in release-1.12.
Best,
Shuiqiang