Sampling rate higher than 1Khz

[Nicholas Walton]

Flinks watermarks are in milliseconds. I have time sampled off a sensor at a rate exceeding 1Khz or 1 per millisecond. Is there a way to handle timestamp granularity below milliseconds, or will I have to generate timestamp for the millisecond value preceding that associated with the sensor reading, which IUC will introduce latency into the processing pipeline.

TIA

[Tzu-Li (Gordon) Tai]

Hi!

Yes, Flink's watermark timestamps are in milliseconds, which means that time-based operators such as time window operators will be fired at a per-millisecond granularity.
Whether or not this introduces "latency" in the pipeline depends on the granularity of your time window operations; if you need to have window durations shorter than 1 millisecond, then yes, having only millisecond watermarks will introduce latency.
Currently in Flink, time-based operations such as windows / registering timers are all done at millisecond accuracy.

Cheers,
Gordon

On Mon, Jan 28, 2019 at 7:55 PM Nicholas Walton <[hidden email]> wrote:

Flinks watermarks are in milliseconds. I have time sampled off a sensor at a rate exceeding 1Khz or 1 per millisecond. Is there a way to handle timestamp granularity below milliseconds, or will I have to generate timestamp for the millisecond value preceding that associated with the sensor reading, which IUC will introduce latency into the processing pipeline.

TIA

[Piotr Nowojski-2]

Hi,

Maybe stupid idea, but does anything prevents a user from pretending that watermarks/event times are in different unit, for example microseconds? Of course assuming using row/event time and not using processing time for anything?

Piotrek 

On 28 Jan 2019, at 14:58, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi!

Yes, Flink's watermark timestamps are in milliseconds, which means that time-based operators such as time window operators will be fired at a per-millisecond granularity.
Whether or not this introduces "latency" in the pipeline depends on the granularity of your time window operations; if you need to have window durations shorter than 1 millisecond, then yes, having only millisecond watermarks will introduce latency.
Currently in Flink, time-based operations such as windows / registering timers are all done at millisecond accuracy.

Cheers,
Gordon

On Mon, Jan 28, 2019 at 7:55 PM Nicholas Walton <[hidden email]> wrote:

Flinks watermarks are in milliseconds. I have time sampled off a sensor at a rate exceeding 1Khz or 1 per millisecond. Is there a way to handle timestamp granularity below milliseconds, or will I have to generate timestamp for the millisecond value preceding that associated with the sensor reading, which IUC will introduce latency into the processing pipeline.

TIA