Tumbling Time Window

[Navneeth Krishnan]

Hello All,

First of all Happy New Year!! Thanks for the excellent community support.

I have a job which requires a 2 seconds tumbling time window per key, For each user we wait for 2 seconds to collect enough data and proceed to

 further processing. My question is should I use the regular DSL windowing or write a custom process function which does the windowing. I have heard that the DSL window has more overhead versus the custom window function.

What do you guys suggest and can someone provide an example of custom window function per key. Also given the window time is very less (2 secs) would there be more overhead in firing so many timers for each key?

Thanks!

Regards,

Navneeth

[Yun Gao]

Hi Navneeth

    For me I think you may start with using the window function and an example for the custom window function could be found in [1]. From the description I think it should be a standard Tumbling window, if implementing with the customized process function, it would end up have a similar functionality with the current window operator. 

Best,

 Yun

 [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/operators/windows.html#processwindowfunction

------------------------------------------------------------------
Sender:Navneeth Krishnan<[hidden email]>
Date:2021/01/04 08:05:17
Recipient:user<[hidden email]>
Theme:Tumbling Time Window

Hello All,

First of all Happy New Year!! Thanks for the excellent community support.

I have a job which requires a 2 seconds tumbling time window per key, For each user we wait for 2 seconds to collect enough data and proceed to

 further processing. My question is should I use the regular DSL windowing or write a custom process function which does the windowing. I have heard that the DSL window has more overhead versus the custom window function.

What do you guys suggest and can someone provide an example of custom window function per key. Also given the window time is very less (2 secs) would there be more overhead in firing so many timers for each key?

Thanks!

Regards,

Navneeth

[David Anderson-3]

For straightforward tumbling windows, the regular DSL windowing performs noticeably better than a custom process function because it takes advantage of an internal API to avoid some serialization overhead. 

There's a simple example of a ProcessWindowFunction in [1], and an example of using a KeyedProcessFunction to do windowing in [2].

Best,

David

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/learn-flink/streaming_analytics.html#window-functions

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/learn-flink/event_driven.html#example

On Mon, Jan 4, 2021 at 1:05 AM Navneeth Krishnan <[hidden email]> wrote:

Hello All,

First of all Happy New Year!! Thanks for the excellent community support.

I have a job which requires a 2 seconds tumbling time window per key, For each user we wait for 2 seconds to collect enough data and proceed to

 further processing. My question is should I use the regular DSL windowing or write a custom process function which does the windowing. I have heard that the DSL window has more overhead versus the custom window function.

What do you guys suggest and can someone provide an example of custom window function per key. Also given the window time is very less (2 secs) would there be more overhead in firing so many timers for each key?

Thanks!

Regards,

Navneeth