Emulate Tumbling window in Event Time Space

[Dhruv Kumar]

Hi

I was trying to emulate tumbling window in event time space. Here is the link to my code.

I am using the process function to do the custom processing which I want to do within every window. I am having an issue of how to emit results at the end of every window since my watermark only gets emitted at every incoming event (incoming event will mostly not intersect with the end time of any window). Seems like I need to add a trigger somewhere which fires at the end of every window. Could any one here help me? Sorry, if I am not clear in anything. I am quite new to Flink. 

Thanks

Dhruv

[Xingcan Cui]

Hi Dhruv,

there’s no need to implement the window logic with the low-level `ProcessFunction` yourself. Flink has provided built-in window operators and you just need to implement the `WindowFunction` for that [1].

Best,

Xingcan

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#window-functions

On 9 Mar 2018, at 1:51 PM, Dhruv Kumar <[hidden email]> wrote:

Hi

I was trying to emulate tumbling window in event time space. Here is the link to my code.

I am using the process function to do the custom processing which I want to do within every window. I am having an issue of how to emit results at the end of every window since my watermark only gets emitted at every incoming event (incoming event will mostly not intersect with the end time of any window). Seems like I need to add a trigger somewhere which fires at the end of every window. Could any one here help me? Sorry, if I am not clear in anything. I am quite new to Flink. 

Thanks

Dhruv

[Piotr Nowojski]

Hi,

As Xingcan responded, you could use already built in operator for that. 

If you really want to implement something on your own (need custom feature? For fun?), you would have to implement some variation of a InternalTimerService from Flink (you can browse the code for an inspiration). On each processed element you have to keep updating state of your in memory/in state windows with timestamps marking when they should be triggered. Then on each processed watermark in your operator you need to trigger/fire windows matching to the processed watermark.

Piotrek

On 9 Mar 2018, at 07:50, Xingcan Cui <[hidden email]> wrote:

Hi Dhruv,

there’s no need to implement the window logic with the low-level `ProcessFunction` yourself. Flink has provided built-in window operators and you just need to implement the `WindowFunction` for that [1].

Best,

Xingcan

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#window-functions

On 9 Mar 2018, at 1:51 PM, Dhruv Kumar <[hidden email]> wrote:

Hi

I was trying to emulate tumbling window in event time space. Here is the link to my code.

I am using the process function to do the custom processing which I want to do within every window. I am having an issue of how to emit results at the end of every window since my watermark only gets emitted at every incoming event (incoming event will mostly not intersect with the end time of any window). Seems like I need to add a trigger somewhere which fires at the end of every window. Could any one here help me? Sorry, if I am not clear in anything. I am quite new to Flink. 

Thanks

Dhruv