Flink Job History Dump

[Robert Schmidtke]

Hi everyone,

I'm using Flink 0.10.2 to run some benchmarks on my cluster and I would like to compare it to Spark 1.6.0. Spark has an eventLog property that I can use to have the history written to HDFS, and then later view it offline on the History Server.

Does Flink have a similar Feature, especially for offline analysis of a job's history/events? I know of the Web UI, but I would like to be able to run my own analysis on top of the data. There is the Monitoring REST API and I'm wondering if it's possible to gain access to the raw data this API exposes, and possibly view it on a locally running web UI.

Thanks a lot in advance

Robert

--

[Ufuk Celebi]

Hey Robert!

This is currently not possible :-(, but this is a feature that is on
Flink's road map.

A very inconvenient workaround could be to manually query the REST
APIs [1] and dump the responses somewhere and query it there.

– Ufuk

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/monitoring_rest_api.html

On Tue, Apr 5, 2016 at 4:00 PM, Robert Schmidtke <[hidden email]> wrote:

> Hi everyone,
>
> I'm using Flink 0.10.2 to run some benchmarks on my cluster and I would like
> to compare it to Spark 1.6.0. Spark has an eventLog property that I can use
> to have the history written to HDFS, and then later view it offline on the
> History Server.
>
> Does Flink have a similar Feature, especially for offline analysis of a
> job's history/events? I know of the Web UI, but I would like to be able to
> run my own analysis on top of the data. There is the Monitoring REST API and
> I'm wondering if it's possible to gain access to the raw data this API
> exposes, and possibly view it on a locally running web UI.
>
> Thanks a lot in advance
> Robert
>
>
> --
> My GPG Key ID: 336E2680