Re: Timestamp and key preservation over operators
Posted by
Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Timestamp-and-key-preservation-over-operators-tp27627p27644.html
Hi,
Actually all operators should preserve record timestamps if set the correct TimeCharacteritics to event time.
A window operator will set the timestamp of all emitted records to the end-timestamp of the window.
Not sure what happens if you use a processing time window in an event time application though...
Can you show a concise example of your program and explain how to check the timestamps?
In general it is not a good idea to assign timestamps and watermarks in the middle of a program because it is can be quite hard to reason about out-of-orderness after the data was shuffled and processed.
You can use reinterpretAsKeyedStream() if an operator does not change the keys and if the parallelism of the source and target operators are the same.
Best,
Fabian
Am Di., 30. Apr. 2019 um 08:59 Uhr schrieb Guowei Ma <
[hidden email]>:
Hi,
Most operators will preserve the input elements timestamp if it has.
Window is a special case. The timestamp of elements emitted by window is the maxTimestamp of the Window which is triggered. Different Window will have different implementation.(GlobalWindow/TimeWindow/CustomizedWindow).
Keyby just shuffle data. I think it could not affect the element's timestamp.
Hope this could help.
Best,
Hello,
I extracted timestamps using BoundedOutOfOrdernessTimestampExtractor from my
sources, have a WindowFunction, and found that my timestamps has been lost.
To do another Window operation, I need to extract timestamp again. I tried
to find a document for that but haven't found one.
Could you please help tell which type of operators would preserve records'
timestamp?
The same question for keyed stream. I have been using the same key
throughout my flow, but with many tranformations (using different operators,
including coProcessFunction, and converting my data between different
classes), and I have been trying to use
DataStreamUtils.reinterpretAsKeyedStream. Is it safe to assume that as long
as I dont do transformation on key, I could use that
reinterpretAsKeyedStream function?
Thanks and best regards,
Averell
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/