Flink metrics missing from UI 1.7.2

[Padarn Wilson-2]

Hi User,

I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job manager separate.

I'm having trouble seeing the metrics from my Flink job in the UI dashboard. Actually I'm using the Datadog reporter to expose most of my metrics, but latency tracking does not seem to be exported.

Is there anything extra that needs to be enabled to make sure metrics are exported and viewable to the Flink UI?

Thanks

[David Anderson-2]

Because latency tracking is expensive, it is turned off by default. You turn it on by setting the interval; that looks something like this:

    env.getConfig().setLatencyTrackingInterval(1000);

The full set of configuration options is described in the docs: https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics

David Anderson | Training Coordinator
Follow us @VervericaData

--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time


On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson <[hidden email]> wrote:

>
> Hi User,
>
> I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job manager separate.
>
> I'm having trouble seeing the metrics from my Flink job in the UI dashboard. Actually I'm using the Datadog reporter to expose most of my metrics, but latency tracking does not seem to be exported.
>
> Is there anything extra that needs to be enabled to make sure metrics are exported and viewable to the Flink UI?
>
> Thanks

[David Anderson-2]

> I have done this (actually I do it in my flink-conf.yaml), but I am not seeing any metrics at all in the Flink UI, 
> let alone the latency tracking. The latency tracking itself does not seem to be exported to datadog (should it be?)

The latency metrics are job metrics, and are not shown in the Flink UI. They are available via the REST API, and I believe they should also be exported to datadog. You will find them at

    http://localhost:8081/jobs/<job id>/metrics

with IDs like

    latency.source_id.bc764cd8ddf7a0cff126f51c16239658.operator_id.ea632d67b7d595e5b851708ae9ad79d6.operator_subtask_index.0.latency_p90

On Sat, Mar 23, 2019 at 1:53 PM Padarn Wilson <[hidden email]> wrote:

Thanks David.

I have done this (actually I do it in my flink-conf.yaml), but I am not seeing any metrics at all in the Flink UI, let alone the latency tracking. The latency tracking itself does not seem to be exported to datadog (should it be?)

On Sat, Mar 23, 2019 at 8:43 PM David Anderson <[hidden email]> wrote:

Because latency tracking is expensive, it is turned off by default. You turn it on by setting the interval; that looks something like this:

    env.getConfig().setLatencyTrackingInterval(1000);

The full set of configuration options is described in the docs: https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics

David Anderson | Training Coordinator
Follow us @VervericaData

--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time


On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson <[hidden email]> wrote:

>
> Hi User,
>
> I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job manager separate.
>
> I'm having trouble seeing the metrics from my Flink job in the UI dashboard. Actually I'm using the Datadog reporter to expose most of my metrics, but latency tracking does not seem to be exported.
>
> Is there anything extra that needs to be enabled to make sure metrics are exported and viewable to the Flink UI?
>
> Thanks

[Padarn Wilson-2]

Thanks David. I cannot see the metrics there, so let me play around a bit more and make sure they are enabled correctly.

On Sat, Mar 23, 2019 at 9:19 PM David Anderson <[hidden email]> wrote:

> I have done this (actually I do it in my flink-conf.yaml), but I am not seeing any metrics at all in the Flink UI, 
> let alone the latency tracking. The latency tracking itself does not seem to be exported to datadog (should it be?)

The latency metrics are job metrics, and are not shown in the Flink UI. They are available via the REST API, and I believe they should also be exported to datadog. You will find them at

    http://localhost:8081/jobs/<job id>/metrics

with IDs like

    latency.source_id.bc764cd8ddf7a0cff126f51c16239658.operator_id.ea632d67b7d595e5b851708ae9ad79d6.operator_subtask_index.0.latency_p90

On Sat, Mar 23, 2019 at 1:53 PM Padarn Wilson <[hidden email]> wrote:

Thanks David.

I have done this (actually I do it in my flink-conf.yaml), but I am not seeing any metrics at all in the Flink UI, let alone the latency tracking. The latency tracking itself does not seem to be exported to datadog (should it be?)

On Sat, Mar 23, 2019 at 8:43 PM David Anderson <[hidden email]> wrote:

Because latency tracking is expensive, it is turned off by default. You turn it on by setting the interval; that looks something like this:

    env.getConfig().setLatencyTrackingInterval(1000);

The full set of configuration options is described in the docs: https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics

David Anderson | Training Coordinator
Follow us @VervericaData

--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time


On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson <[hidden email]> wrote:

>
> Hi User,
>
> I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job manager separate.
>
> I'm having trouble seeing the metrics from my Flink job in the UI dashboard. Actually I'm using the Datadog reporter to expose most of my metrics, but latency tracking does not seem to be exported.
>
> Is there anything extra that needs to be enabled to make sure metrics are exported and viewable to the Flink UI?
>
> Thanks

[dawid]

Padarn Wilson-2 wrote
> I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job
> manager separate.
>
> I'm having trouble seeing the metrics from my Flink job in the UI
> dashboard. Actually I'm using the Datadog reporter to expose most of my
> metrics, but latency tracking does not seem to be exported.
>
> Is there anything extra that needs to be enabled to make sure metrics are
> exported and viewable to the Flink UI?

With Flink 1.7 on Kubernetes you need to make sure the task managers are
registering to the job manager with their IP addresses and not the
hostnames, see the taskmanager-deployment.yaml manifest in [1], with the
K8S_POD_IP environment variable and setting
-Dtaskmanager.host=$(K8S_POD_IP).

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/kubernetes.html#appendix

/David



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Padarn Wilson-2]

Aha! This is almost certainly it. I remembered thinking something like this might be a problem. I'll need to change the deployment a bit to add this (not straightforward to edit the YAML in my case, but thanks!

On Sun, Mar 24, 2019 at 10:01 AM dawid <[hidden email]> wrote:

Padarn Wilson-2 wrote
> I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job
> manager separate.
>
> I'm having trouble seeing the metrics from my Flink job in the UI
> dashboard. Actually I'm using the Datadog reporter to expose most of my
> metrics, but latency tracking does not seem to be exported.
>
> Is there anything extra that needs to be enabled to make sure metrics are
> exported and viewable to the Flink UI?

With Flink 1.7 on Kubernetes you need to make sure the task managers are
registering to the job manager with their IP addresses and not the
hostnames, see the taskmanager-deployment.yaml manifest in [1], with the
K8S_POD_IP environment variable and setting
-Dtaskmanager.host=$(K8S_POD_IP).

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/kubernetes.html#appendix

/David



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/