Apache Flink - Relation between stream time characteristic and timer triggers

[M Singh]

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[Yun Gao]

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[M Singh]

Thanks Yun for your answers.

Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ?  Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?  

Mans

On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <[hidden email]> wrote:

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[Xingcan Cui]

Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)

Cheers,

Xingcan

On Jul 9, 2019, at 11:40 AM, M Singh <[hidden email]> wrote:

Thanks Yun for your answers.

Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ?  Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?  

Mans

On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <[hidden email]> wrote:

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[M Singh]

Thanks for your answer Xingcan.

Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.

Mans

On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <[hidden email]> wrote:

Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)

Cheers,

Xingcan

On Jul 9, 2019, at 11:40 AM, M Singh <[hidden email]> wrote:

Thanks Yun for your answers.

Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ?  Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?  

Mans

On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <[hidden email]> wrote:

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[Fabian Hueske-2]

Hi Mans,

IngestionTime is uses the same internal mechanisms as EventTime (record timestamps and watermarks).

The difference is that instead of extracting a timestamp from the record (using a custom timestamp extractor & wm assigner), Flink will assign timestamps based on the machine clock of the machine that runs the source task and will also automatically generate watermarks. If you ask for my opinion, IngestionTime combines the disadvantages of ProcessingTime and EventTime. You pay the latency / performance penalty of EventTime for the non-determinism of ProcessingTime.

So, if you enable IngestionTime, you can use EventTime timers and ProcessingTime timers.

Best, Fabian

Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <[hidden email]>:

Thanks for your answer Xingcan.

Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.

Mans

On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <[hidden email]> wrote:

Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)

Cheers,

Xingcan

On Jul 9, 2019, at 11:40 AM, M Singh <[hidden email]> wrote:

Thanks Yun for your answers.

Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ?  Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?  

Mans

On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <[hidden email]> wrote:

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[M Singh]

Thanks Fabian for your response.

Just to clarify then - regardless of the time characteristics, if a processor or window trigger registers with a ProcessingTime  and EventTime  timers - they will all fire when the appropriate watermarks arrive.

Thanks again.

On Thursday, July 11, 2019, 05:41:54 AM EDT, Fabian Hueske <[hidden email]> wrote:

Hi Mans,

IngestionTime is uses the same internal mechanisms as EventTime (record timestamps and watermarks).

The difference is that instead of extracting a timestamp from the record (using a custom timestamp extractor & wm assigner), Flink will assign timestamps based on the machine clock of the machine that runs the source task and will also automatically generate watermarks. If you ask for my opinion, IngestionTime combines the disadvantages of ProcessingTime and EventTime. You pay the latency / performance penalty of EventTime for the non-determinism of ProcessingTime.

So, if you enable IngestionTime, you can use EventTime timers and ProcessingTime timers.

Best, Fabian

Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <[hidden email]>:

Thanks for your answer Xingcan.

Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.

Mans

On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <[hidden email]> wrote:

Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)

Cheers,

Xingcan

On Jul 9, 2019, at 11:40 AM, M Singh <[hidden email]> wrote:

Thanks Yun for your answers.

Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ?  Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?  

Mans

On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <[hidden email]> wrote:

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[Fabian Hueske-2]

Hi,

ProcessingTime timers are always supported

EventTime timers are only supported for EventTime and IngestionTime

Best, Fabian

Am Do., 11. Juli 2019 um 17:44 Uhr schrieb M Singh <[hidden email]>:

Thanks Fabian for your response.

Just to clarify then - regardless of the time characteristics, if a processor or window trigger registers with a ProcessingTime  and EventTime  timers - they will all fire when the appropriate watermarks arrive.

Thanks again.

On Thursday, July 11, 2019, 05:41:54 AM EDT, Fabian Hueske <[hidden email]> wrote:

Hi Mans,

IngestionTime is uses the same internal mechanisms as EventTime (record timestamps and watermarks).

The difference is that instead of extracting a timestamp from the record (using a custom timestamp extractor & wm assigner), Flink will assign timestamps based on the machine clock of the machine that runs the source task and will also automatically generate watermarks. If you ask for my opinion, IngestionTime combines the disadvantages of ProcessingTime and EventTime. You pay the latency / performance penalty of EventTime for the non-determinism of ProcessingTime.

So, if you enable IngestionTime, you can use EventTime timers and ProcessingTime timers.

Best, Fabian

Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <[hidden email]>:

Thanks for your answer Xingcan.

Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.

Mans

On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <[hidden email]> wrote:

Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)

Cheers,

Xingcan

On Jul 9, 2019, at 11:40 AM, M Singh <[hidden email]> wrote:

Thanks Yun for your answers.

Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ?  Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?  

Mans

On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <[hidden email]> wrote:

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks

[M Singh]

Thanks Fabian/Xingcan/Yun for all your help.  Mans

On Thursday, July 11, 2019, 11:46:42 AM EDT, Fabian Hueske <[hidden email]> wrote:

Hi,

ProcessingTime timers are always supported

EventTime timers are only supported for EventTime and IngestionTime

Best, Fabian

Am Do., 11. Juli 2019 um 17:44 Uhr schrieb M Singh <[hidden email]>:

Thanks Fabian for your response.

Just to clarify then - regardless of the time characteristics, if a processor or window trigger registers with a ProcessingTime  and EventTime  timers - they will all fire when the appropriate watermarks arrive.

Thanks again.

On Thursday, July 11, 2019, 05:41:54 AM EDT, Fabian Hueske <[hidden email]> wrote:

Hi Mans,

IngestionTime is uses the same internal mechanisms as EventTime (record timestamps and watermarks).

The difference is that instead of extracting a timestamp from the record (using a custom timestamp extractor & wm assigner), Flink will assign timestamps based on the machine clock of the machine that runs the source task and will also automatically generate watermarks. If you ask for my opinion, IngestionTime combines the disadvantages of ProcessingTime and EventTime. You pay the latency / performance penalty of EventTime for the non-determinism of ProcessingTime.

So, if you enable IngestionTime, you can use EventTime timers and ProcessingTime timers.

Best, Fabian

Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <[hidden email]>:

Thanks for your answer Xingcan.

Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.

Mans

On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <[hidden email]> wrote:

Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)

Cheers,

Xingcan

On Jul 9, 2019, at 11:40 AM, M Singh <[hidden email]> wrote:

Thanks Yun for your answers.

Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ?  Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?  

Mans

On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <[hidden email]> wrote:

Hi,

    For the three questions,

  1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.

  2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.

  3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.

Best,

Yun

------------------------------------------------------------------

From:M Singh <[hidden email]>

Send Time:2019 Jul. 9 (Tue.) 09:42

To:User <[hidden email]>

Subject:Apache Flink - Relation between stream time characteristic and timer triggers

Hi:

I have a few questions about the stream time characteristics:

1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?  

2.  Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?

3.  If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using  assignTimeStampAndWatermark to a stream later in the application ?

Thanks