Using latency markers

[Gyula Fóra]

Hi all!

Does anyone have a working example of using the latency markers to test for the topology latency?

We are using Flink 1.3.2 and it seems like however we tune it, whatever job we use all we get is NaN in the metrics. Maybe we are completely missing something...

Thanks!

Gyula

[Aljoscha Krettek]

Hi,

I must admit that I've never used this but I'll try and look into it.

Best,
Aljoscha

> On 10. Aug 2017, at 11:10, Gyula Fóra <[hidden email]> wrote:
>
> Hi all!
>
> Does anyone have a working example of using the latency markers to test for the topology latency?
> We are using Flink 1.3.2 and it seems like however we tune it, whatever job we use all we get is NaN in the metrics. Maybe we are completely missing something...
>
> Thanks!
> Gyula

[Kien Truong]

Hi,

I just want to say we're having the same issues.

There's no metric for latency when we attempted to export the metrics
through graphite either.


Regards,

Kien


On 8/10/2017 7:36 PM, Aljoscha Krettek wrote:

> Hi,
>
> I must admit that I've never used this but I'll try and look into it.
>
> Best,
> Aljoscha
>
>> On 10. Aug 2017, at 11:10, Gyula Fóra <[hidden email]> wrote:
>>
>> Hi all!
>>
>> Does anyone have a working example of using the latency markers to test for the topology latency?
>> We are using Flink 1.3.2 and it seems like however we tune it, whatever job we use all we get is NaN in the metrics. Maybe we are completely missing something...
>>
>> Thanks!
>> Gyula

[Aljoscha Krettek]

It seems you have to enable latency tracking via ExecutionConfig.setLatencyTrackingInterval(...). This will make the sources emit latency tokens, which then in turn update the latency metric: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#latency-tracking.

Did you enable that?

Best,

Aljoscha

On 10. Aug 2017, at 18:36, Kien Truong <[hidden email]> wrote:

Hi,

I just want to say we're having the same issues.

There's no metric for latency when we attempted to export the metrics through graphite either.


Regards,

Kien


On 8/10/2017 7:36 PM, Aljoscha Krettek wrote:

Hi,

I must admit that I've never used this but I'll try and look into it.

Best,
Aljoscha

On 10. Aug 2017, at 11:10, Gyula Fóra <[hidden email]> wrote:

Hi all!

Does anyone have a working example of using the latency markers to test for the topology latency?
We are using Flink 1.3.2 and it seems like however we tune it, whatever job we use all we get is NaN in the metrics. Maybe we are completely missing something...

Thanks!
Gyula

[Gyula Fóra]

Yes, they are enabled by default I think.

Gyula

On Fri, Aug 11, 2017, 14:14 Aljoscha Krettek <[hidden email]> wrote:

It seems you have to enable latency tracking via ExecutionConfig.setLatencyTrackingInterval(...). This will make the sources emit latency tokens, which then in turn update the latency metric: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#latency-tracking.

Did you enable that?

Best,

Aljoscha

On 10. Aug 2017, at 18:36, Kien Truong <[hidden email]> wrote:

Hi,

I just want to say we're having the same issues.

There's no metric for latency when we attempted to export the metrics through graphite either.


Regards,

Kien


On 8/10/2017 7:36 PM, Aljoscha Krettek wrote:

Hi,

I must admit that I've never used this but I'll try and look into it.

Best,
Aljoscha

On 10. Aug 2017, at 11:10, Gyula Fóra <[hidden email]> wrote:

Hi all!

Does anyone have a working example of using the latency markers to test for the topology latency?
We are using Flink 1.3.2 and it seems like however we tune it, whatever job we use all we get is NaN in the metrics. Maybe we are completely missing something...

Thanks!
Gyula

[Kien Truong]

Yes, we also tried changing the tracking interval to no avail, still no latency metric.

Kien

On 8/11/2017 7:26 PM, Gyula Fóra wrote:

Yes, they are enabled by default I think.

Gyula

On Fri, Aug 11, 2017, 14:14 Aljoscha Krettek <[hidden email]> wrote:

It seems you have to enable latency tracking via ExecutionConfig.setLatencyTrackingInterval(...). This will make the sources emit latency tokens, which then in turn update the latency metric: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#latency-tracking.

Did you enable that?

Best,

Aljoscha

On 10. Aug 2017, at 18:36, Kien Truong <[hidden email]> wrote:

Hi,

I just want to say we're having the same issues.

There's no metric for latency when we attempted to export the metrics through graphite either.


Regards,

Kien


On 8/10/2017 7:36 PM, Aljoscha Krettek wrote:

Hi,

I must admit that I've never used this but I'll try and look into it.

Best,
Aljoscha

On 10. Aug 2017, at 11:10, Gyula Fóra <[hidden email]> wrote:

Hi all!

Does anyone have a working example of using the latency markers to test for the topology latency?
We are using Flink 1.3.2 and it seems like however we tune it, whatever job we use all we get is NaN in the metrics. Maybe we are completely missing something...

Thanks!
Gyula

[Aljoscha Krettek]

Ok, I can also confirm that it doesn't work. I'll look into this and will let you know what I find.

Best,

Aljoscha

On 11. Aug 2017, at 14:57, Kien Truong <[hidden email]> wrote:

Yes, we also tried changing the tracking interval to no avail, still no latency metric.

Kien

On 8/11/2017 7:26 PM, Gyula Fóra wrote:

Yes, they are enabled by default I think.

Gyula

On Fri, Aug 11, 2017, 14:14 Aljoscha Krettek <[hidden email]> wrote:

It seems you have to enable latency tracking via ExecutionConfig.setLatencyTrackingInterval(...). This will make the sources emit latency tokens, which then in turn update the latency metric: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#latency-tracking.

Did you enable that?

Best,

Aljoscha

On 10. Aug 2017, at 18:36, Kien Truong <[hidden email]> wrote:

Hi,

I just want to say we're having the same issues.

There's no metric for latency when we attempted to export the metrics through graphite either.


Regards,

Kien


On 8/10/2017 7:36 PM, Aljoscha Krettek wrote:

Hi,

I must admit that I've never used this but I'll try and look into it.

Best,
Aljoscha

On 10. Aug 2017, at 11:10, Gyula Fóra <[hidden email]> wrote:

Hi all!

Does anyone have a working example of using the latency markers to test for the topology latency?
We are using Flink 1.3.2 and it seems like however we tune it, whatever job we use all we get is NaN in the metrics. Maybe we are completely missing something...

Thanks!
Gyula

[aitozi]

Hi, Aljoscha,

the dashboard shown NAN is just because the value of the latencyGague is not
numerical， so it can't be shown in dashboard, i removed the other
latencydescprition except the sink, so i can see the latency in dashboard,
do i need to post a pr?

thanks,
Aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Tzu-Li (Gordon) Tai]

Hi Aitozi,

Yes, I think we haven’t really pin-pointed out the actual cause of the problem, but if you have a fix for that and can provide a PR we can definitely look at it! That would be helpful.

Before opening a PR, also make sure to first open a JIRA for the issue (I don’t think there is one yet for this issue).

Cheers,

Gordon

On 13 September 2017 at 12:14:42 PM, aitozi ([hidden email]) wrote:

Hi, Aljoscha,

the dashboard shown NAN is just because the value of the latencyGague is not
numerical， so it can't be shown in dashboard, i removed the other
latencydescprition except the sink, so i can see the latency in dashboard,
do i need to post a pr?

thanks,
Aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Martin Eden]

Any follow-up on this? Jira? PR?

On Wed, Sep 13, 2017 at 11:30 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Aitozi,

Yes, I think we haven’t really pin-pointed out the actual cause of the problem, but if you have a fix for that and can provide a PR we can definitely look at it! That would be helpful.

Before opening a PR, also make sure to first open a JIRA for the issue (I don’t think there is one yet for this issue).

Cheers,

Gordon

On 13 September 2017 at 12:14:42 PM, aitozi ([hidden email]) wrote:

Hi, Aljoscha,

the dashboard shown NAN is just because the value of the latencyGague is not
numerical， so it can't be shown in dashboard, i removed the other
latencydescprition except the sink, so i can see the latency in dashboard,
do i need to post a pr?

thanks,
Aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Aljoscha Krettek]

This is the Jira issue: https://issues.apache.org/jira/browse/FLINK-7608

On 27. Sep 2017, at 12:22, Martin Eden <[hidden email]> wrote:

Any follow-up on this? Jira? PR?

On Wed, Sep 13, 2017 at 11:30 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Aitozi,

Yes, I think we haven’t really pin-pointed out the actual cause of the problem, but if you have a fix for that and can provide a PR we can definitely look at it! That would be helpful.

Before opening a PR, also make sure to first open a JIRA for the issue (I don’t think there is one yet for this issue).

Cheers,

Gordon

On 13 September 2017 at 12:14:42 PM, aitozi ([hidden email]) wrote:

Hi, Aljoscha,

the dashboard shown NAN is just because the value of the latencyGague is not
numerical， so it can't be shown in dashboard, i removed the other
latencydescprition except the sink, so i can see the latency in dashboard,
do i need to post a pr?

thanks,
Aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Marvin777]

Hi all!

I'm having the same issues. I get NaN in the Task metrics. The version is 1.3.1.

Maybe I'm completely missing something...

There are questions about FLINK-7692, it can Solve the problem above? What should I do in my version.

Thanks!

2017-10-11 23:30 GMT+08:00 Aljoscha Krettek <[hidden email]>:

This is the Jira issue: https://issues.apache.org/jira/browse/FLINK-7608

On 27. Sep 2017, at 12:22, Martin Eden <[hidden email]> wrote:

Any follow-up on this? Jira? PR?

On Wed, Sep 13, 2017 at 11:30 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Aitozi,

Yes, I think we haven’t really pin-pointed out the actual cause of the problem, but if you have a fix for that and can provide a PR we can definitely look at it! That would be helpful.

Before opening a PR, also make sure to first open a JIRA for the issue (I don’t think there is one yet for this issue).

Cheers,

Gordon

On 13 September 2017 at 12:14:42 PM, aitozi ([hidden email]) wrote:

Hi, Aljoscha,

the dashboard shown NAN is just because the value of the latencyGague is not
numerical， so it can't be shown in dashboard, i removed the other
latencydescprition except the sink, so i can see the latency in dashboard,
do i need to post a pr?

thanks,
Aitozi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Tony Wei]

Hi,

I think FLINK-7692 won't solve this problem. It is just a new feature to provide more functionality for user to use metric system.

Regards,

Tony Wei

2017-12-26 18:23 GMT+08:00 Marvin777 <[hidden email]>:

Hi,

I just want to say we're having the same issues. Using latency markers

I get NaN in the Task metrics. The version is 1.3.1.

Maybe I'm completely missing something...

And there are questions about FLINK-7692, it can Solve the problem above? What should I do in my version.

thanks.

[Marvin777]

Hi,

Thanks for your answer.

But now we get NaN in the 'Task Metrics' in the Web UI.

And there's no metric for latency when we attempted to export the metrics through Prometheus either.

Then, we attempted to export the metrics through slf4j, I found that it works. One of the logs is as follows:

"10.203.87.73.taskmanager.e65b594c3a725912566dd631b005c1cc.Flink Streaming RWKafka.Flat Map.0.latency: {LatencySourceDescriptor{vertexID=1, subtaskIndex=-1}={p95=2260.1, min=2.0, mean=247.4375, max=3336.0, p99=3041.069999999994, p50=61.5}}"

What do you think about Flink-7608 and Flink-6053. It is advisable to do so.

Thank you very much.

2017-12-26 18:38 GMT+08:00 Tony Wei <[hidden email]>:

Hi,

I think FLINK-7692 won't solve this problem. It is just a new feature to provide more functionality for user to use metric system.

Regards,

Tony Wei

2017-12-26 18:23 GMT+08:00 Marvin777 <[hidden email]>:

Hi,

I just want to say we're having the same issues. Using latency markers

I get NaN in the Task metrics. The version is 1.3.1.

Maybe I'm completely missing something...

And there are questions about FLINK-7692, it can Solve the problem above? What should I do in my version.

thanks.