Some doubts about window start time and end time

[zhangjun]

,

Hi:

I defined a Tumbling window, I set the time size to one hour, and the resulting windows are [00: 00: 00-01: 00: 00], [01: 00: 00-02: 00: 00]. ....

This meets my expectations, but when I set the time size to 6 hours, the resulting window size is [02: 00: 00-08: 00: 00], [08: 00: 00-14: 00: 00], [14: 00: 00-20: 00: 00] ...

But my expected window size is [00: 00: 00-06: 00: 00], [06: 00: 00-12: 00: 00] ...

Is it right to get such a window result?

thanks

[Caizhi Weng]

Hi Jun,

How do you define your window? Could you please show us the code?

Thanks.

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午5:22写道：

,

Hi:

I defined a Tumbling window, I set the time size to one hour, and the resulting windows are [00: 00: 00-01: 00: 00], [01: 00: 00-02: 00: 00]. ....

This meets my expectations, but when I set the time size to 6 hours, the resulting window size is [02: 00: 00-08: 00: 00], [08: 00: 00-14: 00: 00], [14: 00: 00-20: 00: 00] ...

But my expected window size is [00: 00: 00-06: 00: 00], [06: 00: 00-12: 00: 00] ...

Is it right to get such a window result?

thanks

[zhangjun]

Hi，Caizhi ：

the code like this :

dataStream

.keyBy(“device")

.window(TumblingProcessingTimeWindows.of(Time.hours(6)))

.trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(5)))

                                .aggregate(new MyAggre(), new WindowResultFunction())  

                                 .print();

I add a trigger for quick output

On 11/27/2019 17:54，[hidden email] wrote：

Hi Jun,

How do you define your window? Could you please show us the code?

Thanks.

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午5:22写道：

,

Hi:

I defined a Tumbling window, I set the time size to one hour, and the resulting windows are [00: 00: 00-01: 00: 00], [01: 00: 00-02: 00: 00]. ....

This meets my expectations, but when I set the time size to 6 hours, the resulting window size is [02: 00: 00-08: 00: 00], [08: 00: 00-14: 00: 00], [14: 00: 00-20: 00: 00] ...

But my expected window size is [00: 00: 00-06: 00: 00], [06: 00: 00-12: 00: 00] ...

Is it right to get such a window result?

thanks

[Caizhi Weng]

Hi Jun,

You have to specify an offset when defining the windows. According to the Java docs of TumblingProcessingTimeWindows: "if you are living in somewhere which is not using UTC±00:00 time, such as China which is using UTC+08:00,and you want a time window with size of one day, and window begins at every 00:00:00 of local time,you may use {@code of(Time.days(1),Time.hours(-8))}. The parameter of offset is {@code Time.hours(-8))} since UTC+08:00 is 8 hours earlier than UTC time."

Does this solve the problem?

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午6:03写道：

Hi，Caizhi ：

the code like this :

dataStream

.keyBy(“device")

.window(TumblingProcessingTimeWindows.of(Time.hours(6)))

.trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(5)))

                                .aggregate(new MyAggre(), new WindowResultFunction())  

                                 .print();

I add a trigger for quick output

On 11/27/2019 17:54，[hidden email] wrote：

Hi Jun,

How do you define your window? Could you please show us the code?

Thanks.

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午5:22写道：

,

Hi:

I defined a Tumbling window, I set the time size to one hour, and the resulting windows are [00: 00: 00-01: 00: 00], [01: 00: 00-02: 00: 00]. ....

This meets my expectations, but when I set the time size to 6 hours, the resulting window size is [02: 00: 00-08: 00: 00], [08: 00: 00-14: 00: 00], [14: 00: 00-20: 00: 00] ...

But my expected window size is [00: 00: 00-06: 00: 00], [06: 00: 00-12: 00: 00] ...

Is it right to get such a window result?

thanks

[zhangjun]

Hi,Caizhi :

1.if I add offset , window(TumblingProcessingTimeWindows.of(Time.hours(6),Time.hours(-8)))

   it wil get a error: TumblingProcessingTimeWindows parameters must satisfy abs(offset) < size

2.If it is caused by do not adding an offset, then why the same code, I set the window size to be an hour and there is no problem, and set the window size to six hours will be a problem?

On 11/27/2019 18:21，[hidden email] wrote：

Hi Jun,

You have to specify an offset when defining the windows. According to the Java docs of TumblingProcessingTimeWindows: "if you are living in somewhere which is not using UTC±00:00 time, such as China which is using UTC+08:00,and you want a time window with size of one day, and window begins at every 00:00:00 of local time,you may use {@code of(Time.days(1),Time.hours(-8))}. The parameter of offset is {@code Time.hours(-8))} since UTC+08:00 is 8 hours earlier than UTC time."

Does this solve the problem?

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午6:03写道：

Hi，Caizhi ：

the code like this :

dataStream

.keyBy(“device")

.window(TumblingProcessingTimeWindows.of(Time.hours(6)))

.trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(5)))

                                .aggregate(new MyAggre(), new WindowResultFunction())  

                                 .print();

I add a trigger for quick output

On 11/27/2019 17:54，[hidden email] wrote：

Hi Jun,

How do you define your window? Could you please show us the code?

Thanks.

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午5:22写道：

,

Hi:

I defined a Tumbling window, I set the time size to one hour, and the resulting windows are [00: 00: 00-01: 00: 00], [01: 00: 00-02: 00: 00]. ....

This meets my expectations, but when I set the time size to 6 hours, the resulting window size is [02: 00: 00-08: 00: 00], [08: 00: 00-14: 00: 00], [14: 00: 00-20: 00: 00] ...

But my expected window size is [00: 00: 00-06: 00: 00], [06: 00: 00-12: 00: 00] ...

Is it right to get such a window result?

thanks

[Dawid Wysakowicz-2]

Hi Jun,

First of all how do you actually get the ranges? TimeWindow#getStart returns a long, which you must've interpreted somehow. I will try to describe my suspicions. I think the core problem is that you pass the input in one timezone, but interpret in a different one. (the processing time uses UTC timestamps).  This would explain the shift.

The windows are always aligned in UTC time, with 6h window you get: [-6:00- 0:00], [0:00 - 6:00], [6:00 - 12:00], ... but if you shift it to your local time zone (I assume +8) you get: [2:00-8:00], [8:00, 14:00], ...

If you perform the same operation for 1h window you get (in UTC): [-8:00, -7:00], ..., [-1:00, 0:00],[0:00-1:00], ... and after shifting: [-8:00, -7:00] becomes [0:00 -1:00], which is what you described, but of course it contains different records than the [0:00-1:00] interpreted in UTC.

On the second topic of the offset. The offset actually means an offset within a window. You can shift the window a bit, but you cannot shift the whole timeline. That's why the offset must be smaller than the window size.

Best,

Dawid

On 27/11/2019 11:44, Jun Zhang wrote:

Hi,Caizhi :

1.if I add offset , window(TumblingProcessingTimeWindows.of(Time.hours(6),Time.hours(-8)))

   it wil get a error: TumblingProcessingTimeWindows parameters must satisfy abs(offset) < size

2.If it is caused by do not adding an offset, then why the same code, I set the window size to be an hour and there is no problem, and set the window size to six hours will be a problem?

On 11/27/2019 18:21，[hidden email] wrote：

Hi Jun,

You have to specify an offset when defining the windows. According to the Java docs of TumblingProcessingTimeWindows: "if you are living in somewhere which is not using UTC±00:00 time, such as China which is using UTC+08:00,and you want a time window with size of one day, and window begins at every 00:00:00 of local time,you may use {@code of(Time.days(1),Time.hours(-8))}. The parameter of offset is {@code Time.hours(-8))} since UTC+08:00 is 8 hours earlier than UTC time."

Does this solve the problem?

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午6:03写道：

Hi，Caizhi ：

the code like this :

dataStream

.keyBy(“device")

.window(TumblingProcessingTimeWindows.of(Time.hours(6)))

.trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(5)))

                                .aggregate(new MyAggre(), new WindowResultFunction())  

                                 .print();

I add a trigger for quick output

On 11/27/2019 17:54，[hidden email] wrote：

Hi Jun,

How do you define your window? Could you please show us the code?

Thanks.

Jun Zhang <[hidden email]> 于2019年11月27日周三 下午5:22写道：

,

Hi:

I defined a Tumbling window, I set the time size to one hour, and the resulting windows are [00: 00: 00-01: 00: 00], [01: 00: 00-02: 00: 00]. ....

This meets my expectations, but when I set the time size to 6 hours, the resulting window size is [02: 00: 00-08: 00: 00], [08: 00: 00-14: 00: 00], [14: 00: 00-20: 00: 00] ...

But my expected window size is [00: 00: 00-06: 00: 00], [06: 00: 00-12: 00: 00] ...

Is it right to get such a window result?

thanks