Custom Watermarks with Flink

[Wyatt Frelot]

I originally posted to Stack Overflow because I was trying to figure out he to do this in Flink.

Wondering how to implement something of the sort: 

(1) write up:  https://drive.google.com/file/d/0Bw69DO1tid2_SzVVendtUV9WMVdIUXptQ1hHSl9KNjAyMTBn/view?usp=drivesdk
(2) original post: 

https://stackoverflow.com/questions/50949083/custom-watermarks-with-apache-flink

--

[Fabian Hueske-2]

Hi,

I would not encode this information in watermarks. Watermarks are rather an internal mechanism to reason about event-time.

Flink also generates watermarks internally. This makes the behavior less predictive.

You could either inject special meta data records (which Flink handles just like other annotate each record with the meta data.

Flink 1.5.0 introduced Broadcast State [1] which can be used to apply rules to a full stream, e.g., to set meta data.

Best,

Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/broadcast_state.html

2018-06-23 23:45 GMT+02:00 Wyatt Frelot <[hidden email]>:

I originally posted to Stack Overflow because I was trying to figure out he to do this in Flink.

Wondering how to implement something of the sort: 

(1) write up:  https://drive.google.com/file/d/0Bw69DO1tid2_SzVVendtUV9WMVdIUXptQ1hHSl9KNjAyMTBn/view?usp=drivesdk
(2) original post: 

https://stackoverflow.com/questions/50949083/custom-watermarks-with-apache-flink

--