Measure Latency from source to sink

[Antonio Saldivar Lezama]

Hello

I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.

Thank you

Best Regards

[Hequn Cheng]

Hi antonio,

I see two options to solve your problem.

1. Enable the latency tracking[1]. But you have to pay attention to it's mechanism, for example, a) the sources only periodically emit a special record and b) the latency markers are not accounting for the time user records spend in operators as they are bypassing them.

2. Add a time field to each of your record. Each time a record comes in from the source, write down the time(t1), so that we can get the latency at sink(t2) by  t2 - t1.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#latency-tracking

On Tue, Jun 26, 2018 at 5:23 AM, antonio saldivar <[hidden email]> wrote:

Hello

I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.

Thank you

Best Regards

[Antonio Saldivar Lezama]

Thank you very much

I already did #2 but ate the moment i print te output as i am using a trigger alert and evaluete the window it replace me the toString values to null or 0 and only prints the ones saved in my accumulator and the keyBy value

On Mon, Jun 25, 2018, 9:22 PM Hequn Cheng <[hidden email]> wrote:

Hi antonio,

I see two options to solve your problem.

1. Enable the latency tracking[1]. But you have to pay attention to it's mechanism, for example, a) the sources only periodically emit a special record and b) the latency markers are not accounting for the time user records spend in operators as they are bypassing them.

2. Add a time field to each of your record. Each time a record comes in from the source, write down the time(t1), so that we can get the latency at sink(t2) by  t2 - t1.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#latency-tracking

On Tue, Jun 26, 2018 at 5:23 AM, antonio saldivar <[hidden email]> wrote:

Hello

I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.

Thank you

Best Regards

[Fabian Hueske-2]

Hi,

Measuring latency is tricky and you have to be careful about what you measure.

Aggregations like window operators make things even more difficult because you need to decide which timestamp(s) to forward (smallest?, largest?, all?)

Depending on the operation, the measurement code might even add to the overall latency.

Also, the clock of the nodes in your cluster might not be totally in sync.

Best, Fabian

2018-06-26 4:00 GMT+02:00 antonio saldivar <[hidden email]>:

Thank you very much

I already did #2 but ate the moment i print te output as i am using a trigger alert and evaluete the window it replace me the toString values to null or 0 and only prints the ones saved in my accumulator and the keyBy value

On Mon, Jun 25, 2018, 9:22 PM Hequn Cheng <[hidden email]> wrote:

Hi antonio,

I see two options to solve your problem.

1. Enable the latency tracking[1]. But you have to pay attention to it's mechanism, for example, a) the sources only periodically emit a special record and b) the latency markers are not accounting for the time user records spend in operators as they are bypassing them.

2. Add a time field to each of your record. Each time a record comes in from the source, write down the time(t1), so that we can get the latency at sink(t2) by  t2 - t1.

Hope this helps.

Hequn

[1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#latency-tracking

On Tue, Jun 26, 2018 at 5:23 AM, antonio saldivar <[hidden email]> wrote:

Hello

I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.

Thank you

Best Regards

[zhangminglei]

Hi，Antonio

Usually, the measurement of delay is for specific business I think it is more reasonable. What I understand of latency from my experience is data preparation time plus query calculation time. It is like an end to end latency test. Hopes this can help you. Not point to the latency of flink

Cheers
Minglei


> 在 2018年6月26日，上午5:23，antonio saldivar <[hidden email]> 写道：
>
> Hello
>
> I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.
>
> Thank you
> Best Regards

[Antonio Saldivar Lezama]

Hello Thank you for the feedback,

Well for now I just Want to measure the time that takes form Source to Sink each transaction add the start and end time in mills

El mar., 26 jun. 2018 a las 5:19, zhangminglei (<[hidden email]>) escribió:

Hi，Antonio

Usually, the measurement of delay is for specific business I think it is more reasonable. What I understand of latency from my experience is data preparation time plus query calculation time. It is like an end to end latency test. Hopes this can help you. Not point to the latency of flink

Cheers
Minglei


> 在 2018年6月26日，上午5:23，antonio saldivar <[hidden email]> 写道：
>
> Hello
>
> I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.
>
> Thank you
> Best Regards

[zhangminglei]

Hi, You can do that but it does not makes sense in general. But you can do that by flink, storm, spark streaming or structured streaming.  And make a compare the latency under different framework.

Cheers

Minglei

在 2018年6月26日，下午9:36，antonio saldivar <[hidden email]> 写道：

Hello Thank you for the feedback,

Well for now I just Want to measure the time that takes form Source to Sink each transaction add the start and end time in mills

El mar., 26 jun. 2018 a las 5:19, zhangminglei (<[hidden email]>) escribió:

Hi，Antonio

Usually, the measurement of delay is for specific business I think it is more reasonable. What I understand of latency from my experience is data preparation time plus query calculation time. It is like an end to end latency test. Hopes this can help you. Not point to the latency of flink

Cheers
Minglei


> 在 2018年6月26日，上午5:23，antonio saldivar <[hidden email]> 写道：
>
> Hello
>
> I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.
>
> Thank you
> Best Regards

[Antonio Saldivar Lezama]

Hello thank you

I also was trying using Flink UI Metrics on version 1.4.2  env.getConfig().setLatencyTrackingInterval(1000L), But looks like is not displaying anything

El mar., 26 jun. 2018 a las 10:45, zhangminglei (<[hidden email]>) escribió:

Hi, You can do that but it does not makes sense in general. But you can do that by flink, storm, spark streaming or structured streaming.  And make a compare the latency under different framework.

Cheers

Minglei

在 2018年6月26日，下午9:36，antonio saldivar <[hidden email]> 写道：

Hello Thank you for the feedback,

Well for now I just Want to measure the time that takes form Source to Sink each transaction add the start and end time in mills

El mar., 26 jun. 2018 a las 5:19, zhangminglei (<[hidden email]>) escribió:

Hi，Antonio

Usually, the measurement of delay is for specific business I think it is more reasonable. What I understand of latency from my experience is data preparation time plus query calculation time. It is like an end to end latency test. Hopes this can help you. Not point to the latency of flink

Cheers
Minglei


> 在 2018年6月26日，上午5:23，antonio saldivar <[hidden email]> 写道：
>
> Hello
>
> I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.
>
> Thank you
> Best Regards

[Hequn Cheng]

Hi antonio, latency is exposed via a metric. You can find each operator's latency through flink UI(Overview->Task Metrics -> select the task, for example select the sink -> Add metric -> find latency metric)

On Tue, Jun 26, 2018 at 11:18 PM, antonio saldivar <[hidden email]> wrote:

Hello thank you

I also was trying using Flink UI Metrics on version 1.4.2  env.getConfig().setLatencyTrackingInterval(1000L), But looks like is not displaying anything

El mar., 26 jun. 2018 a las 10:45, zhangminglei (<[hidden email]>) escribió:

Hi, You can do that but it does not makes sense in general. But you can do that by flink, storm, spark streaming or structured streaming.  And make a compare the latency under different framework.

Cheers

Minglei

在 2018年6月26日，下午9:36，antonio saldivar <[hidden email]> 写道：

Hello Thank you for the feedback,

Well for now I just Want to measure the time that takes form Source to Sink each transaction add the start and end time in mills

El mar., 26 jun. 2018 a las 5:19, zhangminglei (<[hidden email]>) escribió:

Hi，Antonio

Usually, the measurement of delay is for specific business I think it is more reasonable. What I understand of latency from my experience is data preparation time plus query calculation time. It is like an end to end latency test. Hopes this can help you. Not point to the latency of flink

Cheers
Minglei


> 在 2018年6月26日，上午5:23，antonio saldivar <[hidden email]> 写道：
>
> Hello
>
> I am trying to measure the latency of each transaction traveling across the system as a DataSource I have a Kafka consumer and I would like to measure the time that takes from the Source to Sink. Does any one has an example?.
>
> Thank you
> Best Regards