Apache Flink Web Dashboard - Completed Job history

[Ovidiu-Cristian MARCU]

Hi

If I restart the Flink I don’t see anymore the history of the completed jobs.
Is this a missing feature or what should I do to see the completed job list history?

Best regards,
Ovidiu

[Ufuk Celebi]

> On 16 Dec 2015, at 15:00, Ovidiu-Cristian MARCU <[hidden email]> wrote:
>
> Hi
>
> If I restart the Flink I don’t see anymore the history of the completed jobs.
> Is this a missing feature or what should I do to see the completed job list history?

Not possible at the moment.

Completed jobs are archived on the JobManager and lost after restarts.

– Ufuk

[Matthias J. Sax-2]

I guess it should be possible to manually save this information from the
corresponding directory and copy it back after restart? But I am not sure?

Please correct me if I am wrong.

-Matthias

On 12/16/2015 03:16 PM, Ufuk Celebi wrote:

>
>> On 16 Dec 2015, at 15:00, Ovidiu-Cristian MARCU <[hidden email]> wrote:
>>
>> Hi
>>
>> If I restart the Flink I don’t see anymore the history of the completed jobs.
>> Is this a missing feature or what should I do to see the completed job list history?
>
> Not possible at the moment.
>
> Completed jobs are archived on the JobManager and lost after restarts.
>
> – Ufuk
>

[Ovidiu-Cristian MARCU]

I would appreciate any indication on how can I save this information at runtime (completed jobs and whatever the Web Dashboard is using) and replay after restart.

I don’t see much information in the web client dirs, configured with
webclient.tempdir: weblog
webclient.uploaddir: weblog/uploaddir
webclient.plandump: weblog/plandump

Best regards,
Ovidiu

> On 16 Dec 2015, at 15:18, Matthias J. Sax <[hidden email]> wrote:
>
> I guess it should be possible to manually save this information from the
> corresponding directory and copy it back after restart? But I am not sure?
>
> Please correct me if I am wrong.
>
> -Matthias
>
> On 12/16/2015 03:16 PM, Ufuk Celebi wrote:
>>
>>> On 16 Dec 2015, at 15:00, Ovidiu-Cristian MARCU <[hidden email]> wrote:
>>>
>>> Hi
>>>
>>> If I restart the Flink I don’t see anymore the history of the completed jobs.
>>> Is this a missing feature or what should I do to see the completed job list history?
>>
>> Not possible at the moment.
>>
>> Completed jobs are archived on the JobManager and lost after restarts.
>>
>> – Ufuk
>>
>

[Fabian Hueske-2]

Hi Ovidiu,

the job execution information is only held in memory and completely discarded when the JobManager is shut down.

However, you can query all stats that are displayed by the dashboard via a REST API [1] while the JM is running and save them yourself.

This way you can analyze the data also after the JM was stopped.

It is not possible to feed that information back into the dashboard.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html

2015-12-16 16:10 GMT+01:00 Ovidiu-Cristian MARCU <[hidden email]>:

I would appreciate any indication on how can I save this information at runtime (completed jobs and whatever the Web Dashboard is using) and replay after restart.

I don’t see much information in the web client dirs, configured with
webclient.tempdir: weblog
webclient.uploaddir: weblog/uploaddir
webclient.plandump: weblog/plandump

Best regards,
Ovidiu

> On 16 Dec 2015, at 15:18, Matthias J. Sax <[hidden email]> wrote:
>
> I guess it should be possible to manually save this information from the
> corresponding directory and copy it back after restart? But I am not sure?
>
> Please correct me if I am wrong.
>
> -Matthias
>
> On 12/16/2015 03:16 PM, Ufuk Celebi wrote:
>>
>>> On 16 Dec 2015, at 15:00, Ovidiu-Cristian MARCU <[hidden email]> wrote:
>>>
>>> Hi
>>>
>>> If I restart the Flink I don’t see anymore the history of the completed jobs.
>>> Is this a missing feature or what should I do to see the completed job list history?
>>
>> Not possible at the moment.
>>
>> Completed jobs are archived on the JobManager and lost after restarts.
>>
>> – Ufuk
>>
>