Within interval for CEP - Wall Clock based or Event Timestamp based?

[Sameer Wadkar]

Hi,

I am using EventTime but when the records get into the CEP PatternStream does the WITHIN interval refer to the wall clock time or the timestamps embedded in the event stream?

If I provide WITHIN(Time.Seconds(5)) and in processing time I am getting events with timestamps in the range of 10 seconds (due to upstream emits), are all those events considered part of the pattern of their timestamps matter?

Thanks,

Sameer

[Maximilian Michels]

Hi Sameer,

That depends on the time characteristic you have chosen. If you have
set it to event time [1] then it will use event time, otherwise the
default is to use processing time.

When using event time, the element's timestamp is used to assign it to
the specified time windows in the patterns, regardless of the time it
actually arrives. The only exception being if the element arrives
after the Watermark, then only the single element will be processed
according to the patterns.

Cheers,
Max

[1] env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);

On Thu, Aug 11, 2016 at 12:39 AM, Sameer W <[hidden email]> wrote:

> Hi,
>
> I am using EventTime but when the records get into the CEP PatternStream
> does the WITHIN interval refer to the wall clock time or the timestamps
> embedded in the event stream?
>
> If I provide WITHIN(Time.Seconds(5)) and in processing time I am getting
> events with timestamps in the range of 10 seconds (due to upstream emits),
> are all those events considered part of the pattern of their timestamps
> matter?
>
> Thanks,
> Sameer

[Sameer Wadkar]

Thanks Max - Especially the last part about late events. 

Sameer

On Fri, Aug 12, 2016 at 4:23 AM, Maximilian Michels <[hidden email]> wrote:

Hi Sameer,

That depends on the time characteristic you have chosen. If you have
set it to event time [1] then it will use event time, otherwise the
default is to use processing time.

When using event time, the element's timestamp is used to assign it to
the specified time windows in the patterns, regardless of the time it
actually arrives. The only exception being if the element arrives
after the Watermark, then only the single element will be processed
according to the patterns.

Cheers,
Max

[1] env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);

On Thu, Aug 11, 2016 at 12:39 AM, Sameer W <[hidden email]> wrote:
> Hi,
>
> I am using EventTime but when the records get into the CEP PatternStream
> does the WITHIN interval refer to the wall clock time or the timestamps
> embedded in the event stream?
>
> If I provide WITHIN(Time.Seconds(5)) and in processing time I am getting
> events with timestamps in the range of 10 seconds (due to upstream emits),
> are all those events considered part of the pattern of their timestamps
> matter?
>
> Thanks,
> Sameer

[Till Rohrmann]

Hi,

at the moment the CEP operator using EventTime only processes elements after it has received a watermark which is larger than the timestamp of the elements. In case of late arrivals it might make sense to directly process these events. This should decrease the latency a bit (for this case).

Cheers,

Till

On Fri, Aug 12, 2016 at 10:29 PM, Sameer W <[hidden email]> wrote:

Thanks Max - Especially the last part about late events. 

Sameer

On Fri, Aug 12, 2016 at 4:23 AM, Maximilian Michels <[hidden email]> wrote:

Hi Sameer,

That depends on the time characteristic you have chosen. If you have
set it to event time [1] then it will use event time, otherwise the
default is to use processing time.

When using event time, the element's timestamp is used to assign it to
the specified time windows in the patterns, regardless of the time it
actually arrives. The only exception being if the element arrives
after the Watermark, then only the single element will be processed
according to the patterns.

Cheers,
Max

[1] env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);

On Thu, Aug 11, 2016 at 12:39 AM, Sameer W <[hidden email]> wrote:
> Hi,
>
> I am using EventTime but when the records get into the CEP PatternStream
> does the WITHIN interval refer to the wall clock time or the timestamps
> embedded in the event stream?
>
> If I provide WITHIN(Time.Seconds(5)) and in processing time I am getting
> events with timestamps in the range of 10 seconds (due to upstream emits),
> are all those events considered part of the pattern of their timestamps
> matter?
>
> Thanks,
> Sameer