Default timestamps for Event Time when no Watermark Assigner used?

[Sameer Wadkar]

Hi,

If you do not explicitly assign timestamps and watermarks when using Event Time, does it automatically default to using Ingestion Time?

I was reading the Kinesis integration section and came across the note below and which raised the above question. I saw another place where you explicitly use Event Time with ingestion time with the following - .assignTimestampsAndWatermarks(new IngestionTimeExtractor<>());. 

Does that line have to called explicitly or is it the default?

"If streaming topologies choose to use the event time notion for record timestamps, an approximate arrival timestamp will be used by default. This timestamp is attached to records by Kinesis once they were successfully received and stored by streams. Note that this timestamp is typically referred to as a Kinesis server-side timestamp, and there are no guarantees about the accuracy or order correctness (i.e., the timestamps may not always be ascending)."

Thanks,

Sameer

[Tzu-Li Tai]

Hi,

For the Kinesis consumer, when you use Event Time but do not explicitly assign timestamps, the Kinesis server-side timestamp (the time which Kinesis received the record) is attached to the record as default, not Flink’s ingestion time.

Does this answer your question?

Regards,

Gordon

On August 23, 2016 at 6:42:26 PM, Sameer W ([hidden email]) wrote:

Hi,

If you do not explicitly assign timestamps and watermarks when using Event Time, does it automatically default to using Ingestion Time?

I was reading the Kinesis integration section and came across the note below and which raised the above question. I saw another place where you explicitly use Event Time with ingestion time with the following - .assignTimestampsAndWatermarks(new IngestionTimeExtractor<>());. 

Does that line have to called explicitly or is it the default?

"If streaming topologies choose to use the event time notion for record timestamps, an approximate arrival timestamp will be used by default. This timestamp is attached to records by Kinesis once they were successfully received and stored by streams. Note that this timestamp is typically referred to as a Kinesis server-side timestamp, and there are no guarantees about the accuracy or order correctness (i.e., the timestamps may not always be ascending)."

Thanks,

Sameer

[Sameer Wadkar]

Thanks - Is there also a default behavior for non Kinesis streams? If I set the time characteristics as Event Time but do not assign timestamps or generate watermarks by invoking the assignTimestampsAndWatermarks function, does that default to using Ingestion time. Or in other words is it like I invoking this method on the source stream-

assignTimestampsAndWatermarks(new IngestionTimeExtractor<>())

Sameer

On Tue, Aug 23, 2016 at 7:29 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi,

For the Kinesis consumer, when you use Event Time but do not explicitly assign timestamps, the Kinesis server-side timestamp (the time which Kinesis received the record) is attached to the record as default, not Flink’s ingestion time.

Does this answer your question?

Regards,

Gordon

On August 23, 2016 at 6:42:26 PM, Sameer W ([hidden email]) wrote:

Hi,

If you do not explicitly assign timestamps and watermarks when using Event Time, does it automatically default to using Ingestion Time?

I was reading the Kinesis integration section and came across the note below and which raised the above question. I saw another place where you explicitly use Event Time with ingestion time with the following - .assignTimestampsAndWatermarks(new IngestionTimeExtractor<>());. 

Does that line have to called explicitly or is it the default?

"If streaming topologies choose to use the event time notion for record timestamps, an approximate arrival timestamp will be used by default. This timestamp is attached to records by Kinesis once they were successfully received and stored by streams. Note that this timestamp is typically referred to as a Kinesis server-side timestamp, and there are no guarantees about the accuracy or order correctness (i.e., the timestamps may not always be ascending)."

Thanks,

Sameer

[Tzu-Li Tai]

No, it does not default to Ingestion Time. For other connectors in general, you have to explicitly call `assignTimestampAndWatermarks()` before the first operator in the topology that works on time (ex. windows), otherwise the job will fail as soon as records start incoming.

Currently, I think only the Kinesis connector and, shortly in the future, Kafka 0.10 connector will have default timestamps when the topology uses Event Time. Otherwise, the behaviour is described as above.

Regards,

Gordon

On August 23, 2016 at 7:34:25 PM, Sameer W ([hidden email]) wrote:

Thanks - Is there also a default behavior for non Kinesis streams? If I set the time characteristics as Event Time but do not assign timestamps or generate watermarks by invoking the assignTimestampsAndWatermarks function, does that default to using Ingestion time. Or in other words is it like I invoking this method on the source stream-

assignTimestampsAndWatermarks(new IngestionTimeExtractor<>())

Sameer

On Tue, Aug 23, 2016 at 7:29 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi,

For the Kinesis consumer, when you use Event Time but do not explicitly assign timestamps, the Kinesis server-side timestamp (the time which Kinesis received the record) is attached to the record as default, not Flink’s ingestion time.

Does this answer your question?

Regards,

Gordon

On August 23, 2016 at 6:42:26 PM, Sameer W ([hidden email]) wrote:

Hi,

If you do not explicitly assign timestamps and watermarks when using Event Time, does it automatically default to using Ingestion Time?

I was reading the Kinesis integration section and came across the note below and which raised the above question. I saw another place where you explicitly use Event Time with ingestion time with the following - .assignTimestampsAndWatermarks(new IngestionTimeExtractor<>());. 

Does that line have to called explicitly or is it the default?

"If streaming topologies choose to use the event time notion for record timestamps, an approximate arrival timestamp will be used by default. This timestamp is attached to records by Kinesis once they were successfully received and stored by streams. Note that this timestamp is typically referred to as a Kinesis server-side timestamp, and there are no guarantees about the accuracy or order correctness (i.e., the timestamps may not always be ascending)."

Thanks,

Sameer