Flink timestamps

[Davood Rafiei]

Hi,

I want to assign timestamps to tuples when they enter source operator ,  get those timestamps in sink operator and extract them from current time and output.

Basically my goal is to measure latency.

To set timestamps in source operator I used:
     
  env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);

However, I don't know how to get the tuple's previously assigned (in source operator) timestamp in sink operator. Is it possible? Or should I insert a separate field to tuple for keeping the timestamp?

Cheers

Davood

[Aljoscha Krettek]

Hi Davood,

right now, you can only inspect the timestamps by writing a custom operator that you would use with DataStream.transform(). Measuring latency this way has some pitfalls, though. The timestamp might be assigned on a different machine than the machine that will process the tuple at the sink/custom operator. If these machines have clocks that are slightly out of sync (as they surely are) then the latency calculation will only be a rough estimate (or completely wrong, in the worst case).

Cheers,

Aljoscha

On Sat, 6 Aug 2016 at 13:47 Davood Rafiei <[hidden email]> wrote:

Hi,

I want to assign timestamps to tuples when they enter source operator ,  get those timestamps in sink operator and extract them from current time and output.

Basically my goal is to measure latency.

To set timestamps in source operator I used:
     
  env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);

However, I don't know how to get the tuple's previously assigned (in source operator) timestamp in sink operator. Is it possible? Or should I insert a separate field to tuple for keeping the timestamp?

Cheers

Davood