Re: Retaining uploaded job jars on Flink HA restarts on Kubernetes

Posted by Rohil Surana on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Retaining-uploaded-job-jars-on-Flink-HA-restarts-on-Kubernetes-tp19985p19996.html

Hey Chirag,

I tried adding both the configs as per the documentation, and I can see the jars getting uploaded to the specified paths, but on JobManager restarts the JARS are actually deleted from the `jobmanager.web.upload.dir` path.

Anything else that I am missing?

Thanks.

- Rohil

On Mon, May 7, 2018 at 11:48 AM, Chirag Dewan <[hidden email]> wrote:

I think you are looking for jobmanager.web.tmpdir along with upload.dir 

From the documentation :

• jobmanager.web.tmpdir: This configuration parameter allows defining the Flink web directory to be used by the web interface. The web interface will copy its static files into the directory. Also uploaded job jars are stored in the directory if not overridden. By default, the temporary directory is used.

• jobmanager.web.upload.dir: The config parameter defining the directory for uploading the job jars. If not specified a dynamic directory will be used under the directory specified by jobmanager.web.tmpdir.

Regards,

Chirag

On Sunday, 6 May, 2018, 12:29:43 AM IST, Rohil Surana <[hidden email]> wrote:

Hi,

I have a very basic Flink HA setup on Kubernetes and wanted to retain job jars on JobManager Restarts.

For HA I am using a Zookeeper and a NFS drive mounted on all pods (JobManager and TaskManagers), that is being used for checkpoints and have also set the `web.upload.dir: /data/flink-uploads` where /data is for the NFS volume.

Still when the JobManager is killed, the uploaded jars are lost.

Would really appreciate if anyone can help in what I am missing.

Here is the link to my flink-conf.yaml - https://pastebin.com/dt7tGTYQ

Thanks.

- Rohil