Re: Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime?

Posted by Aljoscha Krettek on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Do-I-need-to-set-assignTimestampsAndWatermarks-if-I-set-my-time-characteristic-to-IngestionTime-tp33525p33541.html

On 10.03.20 10:13, kant kodali wrote:

> If ingestion time programs cannot handle late data then why would it
> generate watermarks? Isn't the whole point of watermarks is to handle the
> late data?

Watermarks are not only used for handling late data. Watermarks are the
mechanism that is used to update time throughout the streaming topology,
starting from the sources. Among other things is is used to detect late
data.

When setting the characteristic to "ingestion time" you are essentially
instating a watermark extractor that extracts the current processing
time at the sources as event time.

> My last question was more about this library
> <https://github.com/vasia/gelly-streaming> I run several algorithms using
> SimpleEdgeStream.aggregrate(<algoirthm>).print() and I am running into the
> following error whenever I invoke the following constructor
> <https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/SimpleEdgeStream.java#L69>
> .
> But it works if I change it to this
> <https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/SimpleEdgeStream.java#L86>
> so
> I am not exactly sure what is happening there.

I don't know what is going on here, could it be that the library
internally sets the characteristic to event-time, thereby overriding
your ingestion-time setting? In that case you would indeed be missing a
watermark extractor. I'm cc'ing Vasia, as the author of that library.

-Aljoscha