Re: Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime?

Posted by Arvid Heise-3 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Do-I-need-to-set-assignTimestampsAndWatermarks-if-I-set-my-time-characteristic-to-IngestionTime-tp33525p33532.html

Hi Kant,

I just saw that asked the same question on SO [1]. Could you, in the future, please cross-reference these posts, so that we don't waste resources on answering?

[1] https://stackoverflow.com/questions/60610985/do-i-need-to-set-assigntimestampsandwatermarks-if-i-set-my-time-characteristic-t

On Tue, Mar 10, 2020 at 9:33 AM Arvid Heise <[hidden email]> wrote:

Hi Kant,

according to the documentation [1], you don't need to set a watermark assigner:

Compared to event time, ingestion time programs cannot handle any out-of-order events or late data, but the programs don’t have to specify how to generate watermarks.

Internally, ingestion time is treated much like event time, but with automatic timestamp assignment and automatic watermark generation.

So it's neither possible to assign timestamps nor watermark, but it seems as if the default behavior is exactly as you want it to be. If that doesn't work for you, could you please rephrase your last question or describe your use case? I didn't get it.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/event_time.html

On Tue, Mar 10, 2020 at 5:01 AM kant kodali <[hidden email]> wrote:

Hi All,

Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime?

say I set my time characteristic of stream execution environment to Ingestion time as follows

streamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);

do I need to call datastream.assignTimestampsAndWatermarks(AscendingTimestampExtractor) ?

I thought datastream.assignTimestampsAndWatermarks is mandatory only if time characteristic is event time. No? Did this behavior change in Flink 1.10? because I see libraries not setting datastream.assignTimestampsAndWatermarks when time characteristic is Ingestion time but they do for event time. If not, I am wondering how can I set AscendingTimestampExtractor in a distributed environment? is there anyway to add monotonically increasing long(AscendingTimestampExtractor) without any distributed locks?

Thanks!