Completed job wasn't saved to archive
Locked
Completed job wasn't saved to archive
 
|
Hi all, We see occasionally that flink doesn't save information about canceled job to archive directory (configured by jobmanager.archive.fs.dir property). And there are no exceptions in the log about failing archiving. It's a problem in our use case because our script for deploying jobs relies on flink history server to find latest checkpoint for some job. Does flink guarantee saving data to archive? If so, any ideas why it doesn't work sometimes? Flink version is 1.8.0. -- Best regards, Pavel Potseluev Software developer, Yandex.Classifieds LLC |
|
|
If everything is OK(your config options about archive dir and history server is correct), Flink should archive the completed job. You said you did not find any exceptions in the log about failing to archive. But any other exceptions? Can you share the logs about your scene? Best, Vino Pavel Potseluev <[hidden email]> 于2019年11月21日周四 上午2:25写道：
|
Re: Completed job wasn't saved to archive
|
|
Hi Vino, Usually Flink archives jobs correctly and the problem is rarely reproduced. So I think it isn't a problem with configuration. Job Manager log when job 5ec264a20bb5005cdbd8e23a5e59f136 was canceled:
21.11.2019, 06:07, "vino yang" <[hidden email]>:
-- Best regards, Pavel Potseluev Software developer, Yandex.Classifieds LLC |
Re: Completed job wasn't saved to archive
|
|
If the archiving fails there should be
some log message, like "Failed to archive job" or "Could not
archive completed job..." .
If nothing of the sort is logged my
first instinct would be that the operation is being slowed down,
_a lot_.
Where are you archiving them to? Could
it be the write operation is being throttled heavily?
On 21/11/2019 13:48, Pavel Potseluev
wrote:
|
Re: Completed job wasn't saved to archive
|
|
Hi Chesnay, We archive jobs on s3 file system. We don't configure a throttling for write operations and afaik it isn't possible now and will be implemented in FLINK-13251. And other write operations (like checkpoints saving) work fine. But I don't see archived job or message about archiving failure at all. It looks like Flink just didn't try to save job to archive. 21.11.2019, 17:17, "Chesnay Schepler" <[hidden email]>:
-- Best regards, Pavel Potseluev Software developer, Yandex.Classifieds LLC |
Re: Completed job wasn't saved to archive
|
|
I'm afraid I can't think of a solution.
I don't see a way how this operation can succeed or fail without
anything being logged.
Is the cluster behaving normally
afterwards? Could you check whether the numRunningJobs ticks
down properly after the job was canceled?
On 22/11/2019 13:27, Pavel Potseluev
wrote:
|
|
|
Hi Pavel, Sorry for bringing this thread up so late. I was digging into the usage of the Flink history server and I found one situation where there would be no logs and no failure/success message from the cluster: In very rare case in our Flink-YARN session cluster: if an application master (JobManager running container) fails and being restarted as a YARN 2nd attempt (we haven't enable HA) - then there will be no logs of archiving being logged whatsoever. However in this case the there would be a completely new AM container brought up running the JM again (e.g. new log files) I am not exactly sure whether this suites your scenarios. Could you describe a bit more on how your cluster was configured? Thanks, Rong On Mon, Nov 25, 2019 at 10:49 AM Chesnay Schepler <[hidden email]> wrote:
|
| Free forum by Nabble | Edit this page |