How to custom (or use) a window to specify everyday's beginning as watermark?

[Rock]

I need my job to aggregator every device's mertic as daily report.But I did not find a window can cover exactly one day,or let everyday's beginning as watermark .Should I custom a window or any other way to achieve?

[Caizhi Weng]

Hi Rock,

I think you can write your own trigger which fires when the date of the process time of the current record is different from that of the last record.

Pinging @Jark Wu for a more professional answer.

Rock <[hidden email]> 于2019年11月26日周二 下午3:37写道：

I need my job to aggregator every device's mertic as daily report.But I did not find a window can cover exactly one day,or let everyday's beginning as watermark .Should I custom a window or any other way to achieve?

[Jark Wu-3]

Hi Rock,

Sorry, I don't fully understand what you want. 

If you want a tumbling window which covers one day, 

you can use `KeyedStream#timeWindow(Time.days(1))` which covers from UTC 00:00~24:00.

Best,

Jark 

On Tue, 26 Nov 2019 at 17:20, Caizhi Weng <[hidden email]> wrote:

Hi Rock,

I think you can write your own trigger which fires when the date of the process time of the current record is different from that of the last record.

Pinging @Jark Wu for a more professional answer.

Rock <[hidden email]> 于2019年11月26日周二 下午3:37写道：

I need my job to aggregator every device's mertic as daily report.But I did not find a window can cover exactly one day,or let everyday's beginning as watermark .Should I custom a window or any other way to achieve?

[Biao Liu]

Hi Rock,

From my understanding, what you want is a one-day time based window which start at 0 clock.

Actually the one-day time-based window (like Jack mentioned) starts at the beginning of day (0:00). You don't need to do anything special.

If you are using event time window (since you mentioned watermark), the only thing you need to do is generating the correct watermark.

For example, to get the accurate one-day aggregation (you don't want to involve any record from next day), a watermark of 0 clock might be needed. You may find the answer in [1][2].

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/event_timestamps_watermarks.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/event_timestamp_extractors.html

On Tue, 26 Nov 2019 at 17:46, Jark Wu <[hidden email]> wrote:

Hi Rock,

Sorry, I don't fully understand what you want. 

If you want a tumbling window which covers one day, 

you can use `KeyedStream#timeWindow(Time.days(1))` which covers from UTC 00:00~24:00.

Best,

Jark 

On Tue, 26 Nov 2019 at 17:20, Caizhi Weng <[hidden email]> wrote:

Hi Rock,

I think you can write your own trigger which fires when the date of the process time of the current record is different from that of the last record.

Pinging @Jark Wu for a more professional answer.

Rock <[hidden email]> 于2019年11月26日周二 下午3:37写道：

I need my job to aggregator every device's mertic as daily report.But I did not find a window can cover exactly one day,or let everyday's beginning as watermark .Should I custom a window or any other way to achieve?

[Biao Liu]

Hi Rock,

There is an inaccurate description in last response. I don't think a watermark of 0 clock is needed to get the accurate calculation result. The watermark of 0 clock only helps to generate the result you want immediately.

On Tue, 26 Nov 2019 at 18:10, Biao Liu <[hidden email]> wrote:

Hi Rock,

From my understanding, what you want is a one-day time based window which start at 0 clock.

Actually the one-day time-based window (like Jack mentioned) starts at the beginning of day (0:00). You don't need to do anything special.

If you are using event time window (since you mentioned watermark), the only thing you need to do is generating the correct watermark.

For example, to get the accurate one-day aggregation (you don't want to involve any record from next day), a watermark of 0 clock might be needed. You may find the answer in [1][2].

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/event_timestamps_watermarks.html

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/event_timestamp_extractors.html

Thanks,

Biao /'bɪ.aʊ/

On Tue, 26 Nov 2019 at 17:46, Jark Wu <[hidden email]> wrote:

Hi Rock,

Sorry, I don't fully understand what you want. 

If you want a tumbling window which covers one day, 

you can use `KeyedStream#timeWindow(Time.days(1))` which covers from UTC 00:00~24:00.

Best,

Jark 

On Tue, 26 Nov 2019 at 17:20, Caizhi Weng <[hidden email]> wrote:

Hi Rock,

I think you can write your own trigger which fires when the date of the process time of the current record is different from that of the last record.

Pinging @Jark Wu for a more professional answer.

Rock <[hidden email]> 于2019年11月26日周二 下午3:37写道：

I need my job to aggregator every device's mertic as daily report.But I did not find a window can cover exactly one day,or let everyday's beginning as watermark .Should I custom a window or any other way to achieve?