Question about Watermarks within a KeyedProcessFunction

[Marco Villalobos-2]

My source is a Kafka topic.

I am using Event Time.

I assign the event time with an AscendingTimestampExtractor

I noticed when debugging that in the KeyedProcessFunction that

after my highest known event time of:  2020-06-23T00:46:30.000Z

the processElement method had a watermark with an impossible date of:

-292275055-05-16T16:47:04.192Z

but in the onTimer method it had a more reasonable value that trails the highest known event time by 1 millisecond, which is this value:  2020-06-23T00:46:29.999Z

I want to know, why does the processElement method have an impossible watermark value?

[David Anderson-3]

With an AscendingTimestampExtractor, watermarks are not created for every event, and as your job starts up, some events will be processed before the first watermark is generated. 

The impossible value you see is an initial value that's in place until the first real watermark is available. On the other hand, onTimer can not be called until some timer is triggered by the arrival of a watermark, at which point the watermark will have a reasonable value.

On Sat, Jun 27, 2020 at 2:37 AM Marco Villalobos <[hidden email]> wrote:

My source is a Kafka topic.

I am using Event Time.

I assign the event time with an AscendingTimestampExtractor

I noticed when debugging that in the KeyedProcessFunction that

after my highest known event time of:  2020-06-23T00:46:30.000Z

the processElement method had a watermark with an impossible date of:

-292275055-05-16T16:47:04.192Z

but in the onTimer method it had a more reasonable value that trails the highest known event time by 1 millisecond, which is this value:  2020-06-23T00:46:29.999Z

I want to know, why does the processElement method have an impossible watermark value?