Query regarding tumbling event time windows with ingestion time

[Janardhan Reddy]

Hi,

Suppose we have a stream like this.

someStream.timeWindow(Time.minutes(15)).apply {

operation A

}.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {

operation B

}.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {

operation C

}

Say operation A emits some elements => it would be emitted every 15 minutes.

How would be the window behaviour of where operation B takes place if operation A takes more than 5 seconds with ingestion Time characteristic.  Similarly how would windows behave near operation C if operation B takes more than 5 seconds.

Thanks

Janardhan

[Kostas Kloudas]

Hi Janardhan,

After the first windowing operation, the timestamp of the emitted element for each window
will be the (endOfWindow - 1). So in your case, in the second windowing operation (window by 5)
there will be at most one element per window.

I hope this answers your question.

Kostas
 

> On Nov 29, 2016, at 7:25 PM, Janardhan Reddy <[hidden email]> wrote:
>
> Hi,
>
> Suppose we have a stream like this.
>
> someStream.timeWindow(Time.minutes(15)).apply {
> operation A
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation B
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation C
>
> }
>
> Say operation A emits some elements => it would be emitted every 15 minutes.
>
> How would be the window behaviour of where operation B takes place if operation A takes more than 5 seconds with ingestion Time characteristic.  Similarly how would windows behave near operation C if operation B takes more than 5 seconds.
>
>
> Thanks
> Janardhan

[Janardhan Reddy]

HI

i didn't get it , can you please clarify with an example in case each of operation A and B emit multiple elements.

On Wed, Nov 30, 2016 at 3:34 PM, Kostas Kloudas <[hidden email]> wrote:

Hi Janardhan,

After the first windowing operation, the timestamp of the emitted element for each window
will be the (endOfWindow - 1). So in your case, in the second windowing operation (window by 5)
there will be at most one element per window.

I hope this answers your question.

Kostas

> On Nov 29, 2016, at 7:25 PM, Janardhan Reddy <[hidden email]> wrote:
>
> Hi,
>
> Suppose we have a stream like this.
>
> someStream.timeWindow(Time.minutes(15)).apply {
> operation A
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation B
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation C
>
> }
>
> Say operation A emits some elements => it would be emitted every 15 minutes.
>
> How would be the window behaviour of where operation B takes place if operation A takes more than 5 seconds with ingestion Time characteristic.  Similarly how would windows behave near operation C if operation B takes more than 5 seconds.
>
>
> Thanks
> Janardhan

[Kostas Kloudas]

Hello,

If you have events:

A@10m, B@14m, Watermark@15m 

C@16m D@25m, Watermark@30m

then the result will be:

[A,B]@14.59

[C,D]@29.59

This means that for the next windowing, you will have the elements [A,B]@14.59 and [C,D]@29.59

to window in windows of 5min. 

Given that they are 15min apart, this means that you will have:

[A,B]@14.59 in one window and [C,D]@29.59

This is what I meant by at most one element per window, because window 0 to 5 min, 5 to 10 and 

15 to 20 and 20 to 25 will be empty.

Does this make it clearer?

Kostas

On Nov 30, 2016, at 11:48 AM, Janardhan Reddy <[hidden email]> wrote:

HI

i didn't get it , can you please clarify with an example in case each of operation A and B emit multiple elements.

On Wed, Nov 30, 2016 at 3:34 PM, Kostas Kloudas <[hidden email]> wrote:

Hi Janardhan,

After the first windowing operation, the timestamp of the emitted element for each window
will be the (endOfWindow - 1). So in your case, in the second windowing operation (window by 5)
there will be at most one element per window.

I hope this answers your question.

Kostas

> On Nov 29, 2016, at 7:25 PM, Janardhan Reddy <[hidden email]> wrote:
>
> Hi,
>
> Suppose we have a stream like this.
>
> someStream.timeWindow(Time.minutes(15)).apply {
> operation A
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation B
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation C
>
> }
>
> Say operation A emits some elements => it would be emitted every 15 minutes.
>
> How would be the window behaviour of where operation B takes place if operation A takes more than 5 seconds with ingestion Time characteristic.  Similarly how would windows behave near operation C if operation B takes more than 5 seconds.
>
>
> Thanks
> Janardhan

[Janardhan Reddy]

That makes it clear.

Thanks

On Wed, Nov 30, 2016 at 10:15 PM, Kostas Kloudas <[hidden email]> wrote:

Hello,

If you have events:

A@10m, B@14m, Watermark@15m 

C@16m D@25m, Watermark@30m

then the result will be:

[A,B]@14.59

[C,D]@29.59

This means that for the next windowing, you will have the elements [A,B]@14.59 and [C,D]@29.59

to window in windows of 5min. 

Given that they are 15min apart, this means that you will have:

[A,B]@14.59 in one window and [C,D]@29.59

This is what I meant by at most one element per window, because window 0 to 5 min, 5 to 10 and 

15 to 20 and 20 to 25 will be empty.

Does this make it clearer?

Kostas

On Nov 30, 2016, at 11:48 AM, Janardhan Reddy <[hidden email]> wrote:

HI

i didn't get it , can you please clarify with an example in case each of operation A and B emit multiple elements.

On Wed, Nov 30, 2016 at 3:34 PM, Kostas Kloudas <[hidden email]> wrote:

Hi Janardhan,

After the first windowing operation, the timestamp of the emitted element for each window
will be the (endOfWindow - 1). So in your case, in the second windowing operation (window by 5)
there will be at most one element per window.

I hope this answers your question.

Kostas

> On Nov 29, 2016, at 7:25 PM, Janardhan Reddy <[hidden email]> wrote:
>
> Hi,
>
> Suppose we have a stream like this.
>
> someStream.timeWindow(Time.minutes(15)).apply {
> operation A
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation B
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation C
>
> }
>
> Say operation A emits some elements => it would be emitted every 15 minutes.
>
> How would be the window behaviour of where operation B takes place if operation A takes more than 5 seconds with ingestion Time characteristic.  Similarly how would windows behave near operation C if operation B takes more than 5 seconds.
>
>
> Thanks
> Janardhan

[Kostas Kloudas]

No problem! 

Kostas

On Nov 30, 2016, at 7:08 PM, Janardhan Reddy <[hidden email]> wrote:

That makes it clear.

Thanks

On Wed, Nov 30, 2016 at 10:15 PM, Kostas Kloudas <[hidden email]> wrote:

Hello,

If you have events:

A@10m, B@14m, Watermark@15m 

C@16m D@25m, Watermark@30m

then the result will be:

[A,B]@14.59

[C,D]@29.59

This means that for the next windowing, you will have the elements [A,B]@14.59 and [C,D]@29.59

to window in windows of 5min. 

Given that they are 15min apart, this means that you will have:

[A,B]@14.59 in one window and [C,D]@29.59

This is what I meant by at most one element per window, because window 0 to 5 min, 5 to 10 and 

15 to 20 and 20 to 25 will be empty.

Does this make it clearer?

Kostas

On Nov 30, 2016, at 11:48 AM, Janardhan Reddy <[hidden email]> wrote:

HI

i didn't get it , can you please clarify with an example in case each of operation A and B emit multiple elements.

On Wed, Nov 30, 2016 at 3:34 PM, Kostas Kloudas <[hidden email]> wrote:

Hi Janardhan,

After the first windowing operation, the timestamp of the emitted element for each window
will be the (endOfWindow - 1). So in your case, in the second windowing operation (window by 5)
there will be at most one element per window.

I hope this answers your question.

Kostas

> On Nov 29, 2016, at 7:25 PM, Janardhan Reddy <[hidden email]> wrote:
>
> Hi,
>
> Suppose we have a stream like this.
>
> someStream.timeWindow(Time.minutes(15)).apply {
> operation A
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation B
>
> }.keyby("....").window(TumblingEventTimeWindows.ofseconds(5)).apply {
>
> operation C
>
> }
>
> Say operation A emits some elements => it would be emitted every 15 minutes.
>
> How would be the window behaviour of where operation B takes place if operation A takes more than 5 seconds with ingestion Time characteristic.  Similarly how would windows behave near operation C if operation B takes more than 5 seconds.
>
>
> Thanks
> Janardhan