Re: Timestamp and key preservation over operators

Posted by Averell on May 01, 2019; 6:49am
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Timestamp-and-key-preservation-over-operators-tp27627p27658.html

Hi Fabian, Guowei,

I have some updates:
1. I added timestamp&watermark extractor on all of my remaining sources (3 &
4), and the watermark does propagate to my final operator.
2. As I could not find a way to set my file sources as IDLE, I tried to
tweak the class ContinuousFileReaderOperator to be always IDLE:
/ nextElement = format.nextRecord(nextElement);
        if (nextElement != null) {
                readerContext.collect(nextElement);
                if (this.format.getFilePaths()[0].getPath().contains("<myPath>"))
                        readerContext.markAsTemporarilyIdle();
        } else {
/ and the result I got was there's no watermark at all for that stream, and
that IDLE status seemed not to be taken into account (my CEP operator didn't
generate any output). So I do not understand what that IDLE StreamStatus is
for.
My temporary solution, for now, is to use MAX_WATERMARK for those idle
sources. Not sure whether doing that is recommended?

Thanks for your help.
Regards,
Averell





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/