How to get Latency Tracking results?

[Pankaj Chand]

Hello,

How do I visualize (or extract) the results for Latency Tracking for a Flink local cluster? I set "metrics.latency.interval 100" in the conf/flink-conf.yaml file, and started the cluster and SocketWindowWordCount job. However, I could not find the latency distributions anywhere in the web UI, nor are there any latency metrics in the Metrics dropdown box for either task.

I also set "metrics.latency.granularity "operator"" in conf/flink-conf.yaml, but that did not help.

When I tried to query the REST endpoint, I got the following output which did not seem to contain anything related to latency:

$ curl -s http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics

[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]

Thanks,

Pankaj

[David Anderson-3]

Pankaj,

The Flink web UI doesn't do any visualizations of histogram metrics, so the only way to access the latency metrics is either through the REST api or a metrics reporter.

The REST endpoint you tried is the correct place to find these metrics in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6 (if I recall correctly) these metrics were task metrics. So if you are using an older version of Flink you'll need to dig deeper. I believe you'll find them in

/jobs/<jobid>/vertices/<vertexid>/subtasks/metrics

Regards,

David

On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <[hidden email]> wrote:

Hello,

How do I visualize (or extract) the results for Latency Tracking for a Flink local cluster? I set "metrics.latency.interval 100" in the conf/flink-conf.yaml file, and started the cluster and SocketWindowWordCount job. However, I could not find the latency distributions anywhere in the web UI, nor are there any latency metrics in the Metrics dropdown box for either task.

I also set "metrics.latency.granularity "operator"" in conf/flink-conf.yaml, but that did not help.

When I tried to query the REST endpoint, I got the following output which did not seem to contain anything related to latency:

$ curl -s http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics

[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]

Thanks,

Pankaj

[Pankaj Chand]

Hi David, 

Thanks for replying! Sorry, I forgot to mention I am using Flink Version: 1.11.1, Commit ID: 7eb514a.

Is it possible that the default SocketWindowWordCount job is too simple to generate Latency metrics? Or that the latency metrics disappear from the output JSON when the data ingestion is zero?

Thanks,

Pankaj

On Wed, Sep 9, 2020 at 6:27 AM David Anderson <[hidden email]> wrote:

Pankaj,

The Flink web UI doesn't do any visualizations of histogram metrics, so the only way to access the latency metrics is either through the REST api or a metrics reporter.

The REST endpoint you tried is the correct place to find these metrics in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6 (if I recall correctly) these metrics were task metrics. So if you are using an older version of Flink you'll need to dig deeper. I believe you'll find them in

/jobs/<jobid>/vertices/<vertexid>/subtasks/metrics

Regards,

David

On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <[hidden email]> wrote:

Hello,

How do I visualize (or extract) the results for Latency Tracking for a Flink local cluster? I set "metrics.latency.interval 100" in the conf/flink-conf.yaml file, and started the cluster and SocketWindowWordCount job. However, I could not find the latency distributions anywhere in the web UI, nor are there any latency metrics in the Metrics dropdown box for either task.

I also set "metrics.latency.granularity "operator"" in conf/flink-conf.yaml, but that did not help.

When I tried to query the REST endpoint, I got the following output which did not seem to contain anything related to latency:

$ curl -s http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics

[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]

Thanks,

Pankaj

[David Anderson-3]

Pankaj,

I just checked, and the latency metrics for SocketWindowWordCount show up just fine for me with Flink 1.11.1. For me, the latency metrics existed even before I provided any data on the socket for the job to process. This makes sense, as the latency tracking markers will propagate through the job graph and provide measurements despite the lack of any real data for the job to process.

These metrics have horrible names, but for me, 

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics

reveals a bunch of latency metrics, and, for example,

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999

returns

[{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]

Best,

David

On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand <[hidden email]> wrote:

Hi David, 

Thanks for replying! Sorry, I forgot to mention I am using Flink Version: 1.11.1, Commit ID: 7eb514a.

Is it possible that the default SocketWindowWordCount job is too simple to generate Latency metrics? Or that the latency metrics disappear from the output JSON when the data ingestion is zero?

Thanks,

Pankaj

On Wed, Sep 9, 2020 at 6:27 AM David Anderson <[hidden email]> wrote:

Pankaj,

The Flink web UI doesn't do any visualizations of histogram metrics, so the only way to access the latency metrics is either through the REST api or a metrics reporter.

The REST endpoint you tried is the correct place to find these metrics in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6 (if I recall correctly) these metrics were task metrics. So if you are using an older version of Flink you'll need to dig deeper. I believe you'll find them in

/jobs/<jobid>/vertices/<vertexid>/subtasks/metrics

Regards,

David

On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <[hidden email]> wrote:

Hello,

How do I visualize (or extract) the results for Latency Tracking for a Flink local cluster? I set "metrics.latency.interval 100" in the conf/flink-conf.yaml file, and started the cluster and SocketWindowWordCount job. However, I could not find the latency distributions anywhere in the web UI, nor are there any latency metrics in the Metrics dropdown box for either task.

I also set "metrics.latency.granularity "operator"" in conf/flink-conf.yaml, but that did not help.

When I tried to query the REST endpoint, I got the following output which did not seem to contain anything related to latency:

$ curl -s http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics

[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]

Thanks,

Pankaj

[Pankaj Chand]

Thank you, David!

After setting ExecutionConfig for latency tracking in the SocketWindowWordCount Java source code and rebuilding and using that application jar file, I am now getting the latency tracking metrics using the REST endpoint. The below documentation [1] seems to imply that merely setting the parameters in flink-conf.yaml would generate the latency tracking metrics, which is not happening in my case.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking

On Wed, Sep 9, 2020 at 10:08 AM David Anderson <[hidden email]> wrote:

Pankaj,

I just checked, and the latency metrics for SocketWindowWordCount show up just fine for me with Flink 1.11.1. For me, the latency metrics existed even before I provided any data on the socket for the job to process. This makes sense, as the latency tracking markers will propagate through the job graph and provide measurements despite the lack of any real data for the job to process.

These metrics have horrible names, but for me, 

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics

reveals a bunch of latency metrics, and, for example,

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999

returns

[{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]

Best,

David

On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand <[hidden email]> wrote:

Hi David, 

Thanks for replying! Sorry, I forgot to mention I am using Flink Version: 1.11.1, Commit ID: 7eb514a.

Is it possible that the default SocketWindowWordCount job is too simple to generate Latency metrics? Or that the latency metrics disappear from the output JSON when the data ingestion is zero?

Thanks,

Pankaj

On Wed, Sep 9, 2020 at 6:27 AM David Anderson <[hidden email]> wrote:

Pankaj,

The Flink web UI doesn't do any visualizations of histogram metrics, so the only way to access the latency metrics is either through the REST api or a metrics reporter.

The REST endpoint you tried is the correct place to find these metrics in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6 (if I recall correctly) these metrics were task metrics. So if you are using an older version of Flink you'll need to dig deeper. I believe you'll find them in

/jobs/<jobid>/vertices/<vertexid>/subtasks/metrics

Regards,

David

On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <[hidden email]> wrote:

Hello,

How do I visualize (or extract) the results for Latency Tracking for a Flink local cluster? I set "metrics.latency.interval 100" in the conf/flink-conf.yaml file, and started the cluster and SocketWindowWordCount job. However, I could not find the latency distributions anywhere in the web UI, nor are there any latency metrics in the Metrics dropdown box for either task.

I also set "metrics.latency.granularity "operator"" in conf/flink-conf.yaml, but that did not help.

When I tried to query the REST endpoint, I got the following output which did not seem to contain anything related to latency:

$ curl -s http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics

[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]

Thanks,

Pankaj

[Pankaj Chand]

Actually, I was wrong. It turns out I was setting the values the wrong way in conf/flink-conf.yaml.

I set "metrics.latency.interval 100" instead of "metrics.latency.interval: 100".  Sorry about that.

T

On Thu, Sep 10, 2020 at 7:05 AM Pankaj Chand <[hidden email]> wrote:

Thank you, David!

After setting ExecutionConfig for latency tracking in the SocketWindowWordCount Java source code and rebuilding and using that application jar file, I am now getting the latency tracking metrics using the REST endpoint. The below documentation [1] seems to imply that merely setting the parameters in flink-conf.yaml would generate the latency tracking metrics, which is not happening in my case.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking

On Wed, Sep 9, 2020 at 10:08 AM David Anderson <[hidden email]> wrote:

Pankaj,

I just checked, and the latency metrics for SocketWindowWordCount show up just fine for me with Flink 1.11.1. For me, the latency metrics existed even before I provided any data on the socket for the job to process. This makes sense, as the latency tracking markers will propagate through the job graph and provide measurements despite the lack of any real data for the job to process.

These metrics have horrible names, but for me, 

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics

reveals a bunch of latency metrics, and, for example,

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999

returns

[{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]

Best,

David

On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand <[hidden email]> wrote:

Hi David, 

Thanks for replying! Sorry, I forgot to mention I am using Flink Version: 1.11.1, Commit ID: 7eb514a.

Is it possible that the default SocketWindowWordCount job is too simple to generate Latency metrics? Or that the latency metrics disappear from the output JSON when the data ingestion is zero?

Thanks,

Pankaj

On Wed, Sep 9, 2020 at 6:27 AM David Anderson <[hidden email]> wrote:

Pankaj,

The Flink web UI doesn't do any visualizations of histogram metrics, so the only way to access the latency metrics is either through the REST api or a metrics reporter.

The REST endpoint you tried is the correct place to find these metrics in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6 (if I recall correctly) these metrics were task metrics. So if you are using an older version of Flink you'll need to dig deeper. I believe you'll find them in

/jobs/<jobid>/vertices/<vertexid>/subtasks/metrics

Regards,

David

On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <[hidden email]> wrote:

Hello,

How do I visualize (or extract) the results for Latency Tracking for a Flink local cluster? I set "metrics.latency.interval 100" in the conf/flink-conf.yaml file, and started the cluster and SocketWindowWordCount job. However, I could not find the latency distributions anywhere in the web UI, nor are there any latency metrics in the Metrics dropdown box for either task.

I also set "metrics.latency.granularity "operator"" in conf/flink-conf.yaml, but that did not help.

When I tried to query the REST endpoint, I got the following output which did not seem to contain anything related to latency:

$ curl -s http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics

[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]

Thanks,

Pankaj

[David Anderson-3]

Glad you got it working, and thanks for letting us know!

David

On Thu, Sep 10, 2020 at 2:40 PM Pankaj Chand <[hidden email]> wrote:

Actually, I was wrong. It turns out I was setting the values the wrong way in conf/flink-conf.yaml.

I set "metrics.latency.interval 100" instead of "metrics.latency.interval: 100".  Sorry about that.

T

On Thu, Sep 10, 2020 at 7:05 AM Pankaj Chand <[hidden email]> wrote:

Thank you, David!

After setting ExecutionConfig for latency tracking in the SocketWindowWordCount Java source code and rebuilding and using that application jar file, I am now getting the latency tracking metrics using the REST endpoint. The below documentation [1] seems to imply that merely setting the parameters in flink-conf.yaml would generate the latency tracking metrics, which is not happening in my case.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html#latency-tracking

On Wed, Sep 9, 2020 at 10:08 AM David Anderson <[hidden email]> wrote:

Pankaj,

I just checked, and the latency metrics for SocketWindowWordCount show up just fine for me with Flink 1.11.1. For me, the latency metrics existed even before I provided any data on the socket for the job to process. This makes sense, as the latency tracking markers will propagate through the job graph and provide measurements despite the lack of any real data for the job to process.

These metrics have horrible names, but for me, 

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics

reveals a bunch of latency metrics, and, for example,

curl -s http://localhost:8081/jobs/69f954cb93c680fb2b210f877c377131/metrics?get=latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999

returns

[{"id":"latency.source_id.cbc357ccb763df2852fee8c4fc7d55f2.operator_id.17fbfcaabad45985bbdf4da0490487e3.operator_subtask_index.0.latency_p999","value":"105.0"}]

Best,

David

On Wed, Sep 9, 2020 at 3:16 PM Pankaj Chand <[hidden email]> wrote:

Hi David, 

Thanks for replying! Sorry, I forgot to mention I am using Flink Version: 1.11.1, Commit ID: 7eb514a.

Is it possible that the default SocketWindowWordCount job is too simple to generate Latency metrics? Or that the latency metrics disappear from the output JSON when the data ingestion is zero?

Thanks,

Pankaj

On Wed, Sep 9, 2020 at 6:27 AM David Anderson <[hidden email]> wrote:

Pankaj,

The Flink web UI doesn't do any visualizations of histogram metrics, so the only way to access the latency metrics is either through the REST api or a metrics reporter.

The REST endpoint you tried is the correct place to find these metrics in all recent versions of Flink, but somewhere back before Flink 1.5 or 1.6 (if I recall correctly) these metrics were task metrics. So if you are using an older version of Flink you'll need to dig deeper. I believe you'll find them in

/jobs/<jobid>/vertices/<vertexid>/subtasks/metrics

Regards,

David

On Tue, Sep 8, 2020 at 10:52 PM Pankaj Chand <[hidden email]> wrote:

Hello,

How do I visualize (or extract) the results for Latency Tracking for a Flink local cluster? I set "metrics.latency.interval 100" in the conf/flink-conf.yaml file, and started the cluster and SocketWindowWordCount job. However, I could not find the latency distributions anywhere in the web UI, nor are there any latency metrics in the Metrics dropdown box for either task.

I also set "metrics.latency.granularity "operator"" in conf/flink-conf.yaml, but that did not help.

When I tried to query the REST endpoint, I got the following output which did not seem to contain anything related to latency:

$ curl -s http://localhost:8081/jobs/5d0e348eb68588646dece3654d846cf3/metrics

[{"id":"numberOfFailedCheckpoints"},{"id":"lastCheckpointSize"},{"id":"lastCheckpointExternalPath"},{"id":"totalNumberOfCheckpoints"},{"id":"lastCheckpointRestoreTimestamp"},{"id":"uptime"},{"id":"restartingTime"},{"id":"numberOfInProgressCheckpoints"},{"id":"downtime"},{"id":"numberOfCompletedCheckpoints"},{"id":"numRestarts"},{"id":"fullRestarts"},{"id":"lastCheckpointDuration"}]

Thanks,

Pankaj