History Server

[Elias Levy]

I am curious, why is the History Server a separate process and Web UI instead of being part of the Web Dashboard within the Job Manager?

[Nico Kruber]

Hi Elias,
in theory, it could be integrated into a single web interface, but this was
not done so far.
I guess the main reason for keeping it separate was probably to have a better
separation of concerns as the history server is actually independent of the
current JobManager execution and merely displays previous job results which
may also come from different or previously existing JobManager instances which
stored history data in its storage directory.

Chesnay (cc'd) may elaborate a bit more in case you'd like to change that and
integrate the history server (interface) into the JobManager.


Nico

On Sunday, 24 September 2017 02:48:40 CEST Elias Levy wrote:
> I am curious, why is the History Server a separate process and Web UI
> instead of being part of the Web Dashboard within the Job Manager?

[Stephan Ewen]

To add to this:

The History Server is mainly useful in cases where one runs a Flink-cluster-per-job. One the job finished, the processes disappear. The History Server should be longer lived to make past executions' stats available.

On Mon, Sep 25, 2017 at 3:44 PM, Nico Kruber <[hidden email]> wrote:

Hi Elias,
in theory, it could be integrated into a single web interface, but this was
not done so far.
I guess the main reason for keeping it separate was probably to have a better
separation of concerns as the history server is actually independent of the
current JobManager execution and merely displays previous job results which
may also come from different or previously existing JobManager instances which
stored history data in its storage directory.

Chesnay (cc'd) may elaborate a bit more in case you'd like to change that and
integrate the history server (interface) into the JobManager.


Nico

On Sunday, 24 September 2017 02:48:40 CEST Elias Levy wrote:
> I am curious, why is the History Server a separate process and Web UI
> instead of being part of the Web Dashboard within the Job Manager?

[Eron Wright]

As a follow-up question, how well does the history server work for observing a running job?   I'm trying to understand whether, in the cluster-per-job model, a user would be expected to hop from the Web UI to the History Server once the job completed.

Thanks

On Wed, Oct 4, 2017 at 3:49 AM, Stephan Ewen <[hidden email]> wrote:

To add to this:

The History Server is mainly useful in cases where one runs a Flink-cluster-per-job. One the job finished, the processes disappear. The History Server should be longer lived to make past executions' stats available.

On Mon, Sep 25, 2017 at 3:44 PM, Nico Kruber <[hidden email]> wrote:

Hi Elias,
in theory, it could be integrated into a single web interface, but this was
not done so far.
I guess the main reason for keeping it separate was probably to have a better
separation of concerns as the history server is actually independent of the
current JobManager execution and merely displays previous job results which
may also come from different or previously existing JobManager instances which
stored history data in its storage directory.

Chesnay (cc'd) may elaborate a bit more in case you'd like to change that and
integrate the history server (interface) into the JobManager.


Nico

On Sunday, 24 September 2017 02:48:40 CEST Elias Levy wrote:
> I am curious, why is the History Server a separate process and Web UI
> instead of being part of the Web Dashboard within the Job Manager?

[Chesnay Schepler]

The HS isn't aware of running jobs at all, so yes users would have to switch once the job is done.

On 17.01.2018 00:45, Eron Wright wrote:

As a follow-up question, how well does the history server work for observing a running job?   I'm trying to understand whether, in the cluster-per-job model, a user would be expected to hop from the Web UI to the History Server once the job completed.

Thanks

On Wed, Oct 4, 2017 at 3:49 AM, Stephan Ewen <[hidden email]> wrote:

To add to this:

The History Server is mainly useful in cases where one runs a Flink-cluster-per-job. One the job finished, the processes disappear. The History Server should be longer lived to make past executions' stats available.

On Mon, Sep 25, 2017 at 3:44 PM, Nico Kruber <[hidden email]> wrote:

Hi Elias,
in theory, it could be integrated into a single web interface, but this was
not done so far.
I guess the main reason for keeping it separate was probably to have a better
separation of concerns as the history server is actually independent of the
current JobManager execution and merely displays previous job results which
may also come from different or previously existing JobManager instances which
stored history data in its storage directory.

Chesnay (cc'd) may elaborate a bit more in case you'd like to change that and
integrate the history server (interface) into the JobManager.


Nico

On Sunday, 24 September 2017 02:48:40 CEST Elias Levy wrote:
> I am curious, why is the History Server a separate process and Web UI
> instead of being part of the Web Dashboard within the Job Manager?