What's the best practice to determine whether a job has finished or not?

[Caizhi Weng]

Hi dear Flink community,

I would like to determine whether a job has finished (no matter successfully or exceptionally) in my code.

I used to think that JobClient#getJobStatus is a good idea, but I found that it behaves quite differently under different executing environments. For example, under a standalone session cluster it will return the FINISHED status for a finished job, while under a yarn per job cluster it will throw a ApplicationNotFound exception. I'm afraid that there might be other behaviors for other environments.

So what's the best practice to determine whether a job has finished or not? Note that I'm not waiting for the job to finish. If the job hasn't finished I would like to know it and do something else.

[Jeff Zhang]

I use JobListener#onJobExecuted to be notified that the flink job is done.

It is pretty reliable for me, the only exception is the client process is down. 

BTW, the reason you see ApplicationNotFound exception is that yarn app is terminated which means the flink cluster is shutdown. While for standalone mode, the flink cluster is always up.

Caizhi Weng <[hidden email]> 于2020年5月8日周五 下午2:47写道：

Hi dear Flink community,

I would like to determine whether a job has finished (no matter successfully or exceptionally) in my code.

I used to think that JobClient#getJobStatus is a good idea, but I found that it behaves quite differently under different executing environments. For example, under a standalone session cluster it will return the FINISHED status for a finished job, while under a yarn per job cluster it will throw a ApplicationNotFound exception. I'm afraid that there might be other behaviors for other environments.

So what's the best practice to determine whether a job has finished or not? Note that I'm not waiting for the job to finish. If the job hasn't finished I would like to know it and do something else.

--

[Caizhi Weng]

Hi Jeff,

Thanks for the response. However I'm using executeAsync so that I can run the job asynchronously and get a JobClient to monitor the job. JobListener only works for synchronous execute method. Is there other way to achieve this?

Jeff Zhang <[hidden email]> 于2020年5月8日周五 下午3:29写道：

I use JobListener#onJobExecuted to be notified that the flink job is done.

It is pretty reliable for me, the only exception is the client process is down. 

BTW, the reason you see ApplicationNotFound exception is that yarn app is terminated which means the flink cluster is shutdown. While for standalone mode, the flink cluster is always up.

Caizhi Weng <[hidden email]> 于2020年5月8日周五 下午2:47写道：

Hi dear Flink community,

I would like to determine whether a job has finished (no matter successfully or exceptionally) in my code.

I used to think that JobClient#getJobStatus is a good idea, but I found that it behaves quite differently under different executing environments. For example, under a standalone session cluster it will return the FINISHED status for a finished job, while under a yarn per job cluster it will throw a ApplicationNotFound exception. I'm afraid that there might be other behaviors for other environments.

So what's the best practice to determine whether a job has finished or not? Note that I'm not waiting for the job to finish. If the job hasn't finished I would like to know it and do something else.

--

Best Regards

Jeff Zhang

[Kurt Young]

[hidden email] 

On Fri, May 8, 2020 at 3:35 PM Caizhi Weng <[hidden email]> wrote:

Hi Jeff,

Thanks for the response. However I'm using executeAsync so that I can run the job asynchronously and get a JobClient to monitor the job. JobListener only works for synchronous execute method. Is there other way to achieve this?

Jeff Zhang <[hidden email]> 于2020年5月8日周五 下午3:29写道：

I use JobListener#onJobExecuted to be notified that the flink job is done.

It is pretty reliable for me, the only exception is the client process is down. 

BTW, the reason you see ApplicationNotFound exception is that yarn app is terminated which means the flink cluster is shutdown. While for standalone mode, the flink cluster is always up.

Caizhi Weng <[hidden email]> 于2020年5月8日周五 下午2:47写道：

Hi dear Flink community,

I would like to determine whether a job has finished (no matter successfully or exceptionally) in my code.

I used to think that JobClient#getJobStatus is a good idea, but I found that it behaves quite differently under different executing environments. For example, under a standalone session cluster it will return the FINISHED status for a finished job, while under a yarn per job cluster it will throw a ApplicationNotFound exception. I'm afraid that there might be other behaviors for other environments.

So what's the best practice to determine whether a job has finished or not? Note that I'm not waiting for the job to finish. If the job hasn't finished I would like to know it and do something else.

--

Best Regards

Jeff Zhang