(DEPRECATED) Apache Flink User Mailing List archive.

latency monitoring

classic Classic list List threaded Threaded
♦
♦
 3 messages Options
Fanbin Bu
Reply | Threaded
Open this post in threaded view
♦
♦
|

latency monitoring

Fanbin Bu
Hi,

I would like to monitor the pipeline latency measured by 
timestamp when writing the output to sink - timestamp when ingested from the source.

Now I'm able to get the timestamp writing to sink since the sink is implementing a RichSinkFunction and therefore I can report gauge there [1]. But I have no idea on how to get the source ingestion timestamp since I use Flink SQL create table DDL [2] to create a table and use Flink SQL for the logic.

I also checked [3] which is not recommended for prod use cases.

Any suggestions?
Thanks,
Fanbin
Jaffe, Julian
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: latency monitoring

Jaffe, Julian

Hey Fanbin,

 

I’m not sure if TimeCharacteristic.IngestionTime works with Flink SQL, but if you haven’t tried setting the stream time characteristic to ingestion time it’s worth a shot. Otherwise, one possibility that comes to mind is to use a custom TimestampAssigner to set the event time to the ingestion time (this could be as simple as just returning System.currentTimeMillis() in extractAscendingTimestamp). Then in your sink you can compare the current processing time to the event’s timestamp and report the latency.

 

Julian

 

From: Fanbin Bu <[hidden email]>
Date: Thursday, December 10, 2020 at 3:41 PM
To: user <[hidden email]>
Subject: latency monitoring

 

Hi,

 

I would like to monitor the pipeline latency measured by 

timestamp when writing the output to sink - timestamp when ingested from the source.

 

Now I'm able to get the timestamp writing to sink since the sink is implementing a RichSinkFunction and therefore I can report gauge there [1]. But I have no idea on how to get the source ingestion timestamp since I use Flink SQL create table DDL [2] to create a table and use Flink SQL for the logic.

 

I also checked [3] which is not recommended for prod use cases.

 

Any suggestions?

Thanks,

Fanbin

 
Fanbin Bu
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: latency monitoring

Fanbin Bu
1) i have to set the time characteristic to event time
2) I use Flink SQL create table DDL: create table a (user_id string, ts timestamp, data string) . so the timestampAssigner is automatically setup using the `ts` field.

Still not sure how to get the source ingestion time to metrics system.



On Thu, Dec 10, 2020 at 5:48 PM Jaffe, Julian <[hidden email]> wrote:

Hey Fanbin,

 

I’m not sure if TimeCharacteristic.IngestionTime works with Flink SQL, but if you haven’t tried setting the stream time characteristic to ingestion time it’s worth a shot. Otherwise, one possibility that comes to mind is to use a custom TimestampAssigner to set the event time to the ingestion time (this could be as simple as just returning System.currentTimeMillis() in extractAscendingTimestamp). Then in your sink you can compare the current processing time to the event’s timestamp and report the latency.

 

Julian

 

From: Fanbin Bu <[hidden email]>
Date: Thursday, December 10, 2020 at 3:41 PM
To: user <[hidden email]>
Subject: latency monitoring

 

Hi,

 

I would like to monitor the pipeline latency measured by 

timestamp when writing the output to sink - timestamp when ingested from the source.

 

Now I'm able to get the timestamp writing to sink since the sink is implementing a RichSinkFunction and therefore I can report gauge there [1]. But I have no idea on how to get the source ingestion timestamp since I use Flink SQL create table DDL [2] to create a table and use Flink SQL for the logic.

 

I also checked [3] which is not recommended for prod use cases.

 

Any suggestions?

Thanks,

Fanbin

 
Free forum by Nabble Edit this page