Re: Timestamp and key preservation over operators
Posted by
Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Timestamp-and-key-preservation-over-operators-tp27627p27669.html
Hi Averell,
The watermark of a stream is always the low watermark of all its input streams. If one of the input streams does not have watermarks, Flink does not compute a watermark for the merged stream.
If you do not need time-based operations on streams 3 and 4, setting the watermark to MAX_WATERMARK should be a good solution.
Best, Fabian
Hi Fabian, Guowei,
I have some updates:
1. I added timestamp&watermark extractor on all of my remaining sources (3 &
4), and the watermark does propagate to my final operator.
2. As I could not find a way to set my file sources as IDLE, I tried to
tweak the class ContinuousFileReaderOperator to be always IDLE:
/ nextElement = format.nextRecord(nextElement);
if (nextElement != null) {
readerContext.collect(nextElement);
if (this.format.getFilePaths()[0].getPath().contains("<myPath>"))
readerContext.markAsTemporarilyIdle();
} else {
/ and the result I got was there's no watermark at all for that stream, and
that IDLE status seemed not to be taken into account (my CEP operator didn't
generate any output). So I do not understand what that IDLE StreamStatus is
for.
My temporary solution, for now, is to use MAX_WATERMARK for those idle
sources. Not sure whether doing that is recommended?
Thanks for your help.
Regards,
Averell
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/