implementing a continuous time window

[Jonathan Yom-Tov]

hi,

Is it possible to implement a continuous time window with flink? Here's an
example. Say I want to count events within a window. The window length is 5
seconds and I get events at t = 1, 2, 7, 8 seconds. I would then expect to
get events with a count at t = 1 (count = 1), t = 2 (count = 2), t = 6
(count = 1), t = 7 (count = 2), t = 8 (count = 2), t = 12 (count = 1) and t
= 13 (count = 0).

How would I go about doing that?.

thanks,
Jon.

[John Sherwood]

You are looking for sliding windows: https://flink.apache.org/news/2015/12/04/Introducing-windows.html

Here you would do

.timeWindow(Time.seconds(5), Time.seconds(1))

On Thu, Apr 21, 2016 at 12:06 PM, Jonathan Yom-Tov <[hidden email]> wrote:

hi,

Is it possible to implement a continuous time window with flink? Here's an
example. Say I want to count events within a window. The window length is 5
seconds and I get events at t = 1, 2, 7, 8 seconds. I would then expect to
get events with a count at t = 1 (count = 1), t = 2 (count = 2), t = 6
(count = 1), t = 7 (count = 2), t = 8 (count = 2), t = 12 (count = 1) and t
= 13 (count = 0).

How would I go about doing that?.

thanks,
Jon.

[Jonathan Yom-Tov]

I think sliding windows are different. In the example in the blog post a window is computed every 30 seconds (so at fixed time intervals). What I want is for a window to be computed every time an event comes in and then once again when the event leaves the window.

On Thu, Apr 21, 2016 at 10:14 PM, John Sherwood <[hidden email]> wrote:

You are looking for sliding windows: https://flink.apache.org/news/2015/12/04/Introducing-windows.html

Here you would do

.timeWindow(Time.seconds(5), Time.seconds(1))

On Thu, Apr 21, 2016 at 12:06 PM, Jonathan Yom-Tov <[hidden email]> wrote:

hi,

Is it possible to implement a continuous time window with flink? Here's an
example. Say I want to count events within a window. The window length is 5
seconds and I get events at t = 1, 2, 7, 8 seconds. I would then expect to
get events with a count at t = 1 (count = 1), t = 2 (count = 2), t = 6
(count = 1), t = 7 (count = 2), t = 8 (count = 2), t = 12 (count = 1) and t
= 13 (count = 0).

How would I go about doing that?.

thanks,
Jon.

[Fabian Hueske-2]

Yes, sliding windows are different.

You want to evaluate the window whenever a new element arrives or an element leaves because 5 secs passed since it entered the window, right?

I think that should be possible with a GlobalWindow, a custom Trigger which holds state about the time when each element in the window entered the window, and an Evictor.

2016-04-21 21:19 GMT+02:00 Jonathan Yom-Tov <[hidden email]>:

I think sliding windows are different. In the example in the blog post a window is computed every 30 seconds (so at fixed time intervals). What I want is for a window to be computed every time an event comes in and then once again when the event leaves the window.

On Thu, Apr 21, 2016 at 10:14 PM, John Sherwood <[hidden email]> wrote:

You are looking for sliding windows: https://flink.apache.org/news/2015/12/04/Introducing-windows.html

Here you would do

.timeWindow(Time.seconds(5), Time.seconds(1))

On Thu, Apr 21, 2016 at 12:06 PM, Jonathan Yom-Tov <[hidden email]> wrote:

hi,

Is it possible to implement a continuous time window with flink? Here's an
example. Say I want to count events within a window. The window length is 5
seconds and I get events at t = 1, 2, 7, 8 seconds. I would then expect to
get events with a count at t = 1 (count = 1), t = 2 (count = 2), t = 6
(count = 1), t = 7 (count = 2), t = 8 (count = 2), t = 12 (count = 1) and t
= 13 (count = 0).

How would I go about doing that?.

thanks,
Jon.

[Jonathan Yom-Tov]

Thanks. Any pointers on how to do that? Or code examples which do similar things?

On Thu, Apr 21, 2016 at 10:30 PM, Fabian Hueske <[hidden email]> wrote:

Yes, sliding windows are different.

You want to evaluate the window whenever a new element arrives or an element leaves because 5 secs passed since it entered the window, right?

I think that should be possible with a GlobalWindow, a custom Trigger which holds state about the time when each element in the window entered the window, and an Evictor.

2016-04-21 21:19 GMT+02:00 Jonathan Yom-Tov <[hidden email]>:

I think sliding windows are different. In the example in the blog post a window is computed every 30 seconds (so at fixed time intervals). What I want is for a window to be computed every time an event comes in and then once again when the event leaves the window.

On Thu, Apr 21, 2016 at 10:14 PM, John Sherwood <[hidden email]> wrote:

You are looking for sliding windows: https://flink.apache.org/news/2015/12/04/Introducing-windows.html

Here you would do

.timeWindow(Time.seconds(5), Time.seconds(1))

On Thu, Apr 21, 2016 at 12:06 PM, Jonathan Yom-Tov <[hidden email]> wrote:

hi,

Is it possible to implement a continuous time window with flink? Here's an
example. Say I want to count events within a window. The window length is 5
seconds and I get events at t = 1, 2, 7, 8 seconds. I would then expect to
get events with a count at t = 1 (count = 1), t = 2 (count = 2), t = 6
(count = 1), t = 7 (count = 2), t = 8 (count = 2), t = 12 (count = 1) and t
= 13 (count = 0).

How would I go about doing that?.

thanks,
Jon.

[Fabian Hueske-2]

Hi Jonathan,

I thought about your use case again. I'm afraid, the approach I proposed is not working due to limitations of the Evictor interface.

The only way that I see to implement you use case is to implement a custom stream operator by extending AbstractStreamOperator and implementing the OneInputStreamOperator interface.

The operator is called for each arriving element and offers timed call-backs. You would have to take care of buffering the elements, registering timers, and emitting elements yourself.

If you do that, you should make sure that all state is kept in Flink's managed state to make sure that your operator can recover from failures.

Cheers, Fabian

2016-04-21 23:16 GMT+02:00 Jonathan Yom-Tov <[hidden email]>:

Thanks. Any pointers on how to do that? Or code examples which do similar things?

On Thu, Apr 21, 2016 at 10:30 PM, Fabian Hueske <[hidden email]> wrote:

Yes, sliding windows are different.

You want to evaluate the window whenever a new element arrives or an element leaves because 5 secs passed since it entered the window, right?

I think that should be possible with a GlobalWindow, a custom Trigger which holds state about the time when each element in the window entered the window, and an Evictor.

2016-04-21 21:19 GMT+02:00 Jonathan Yom-Tov <[hidden email]>:

I think sliding windows are different. In the example in the blog post a window is computed every 30 seconds (so at fixed time intervals). What I want is for a window to be computed every time an event comes in and then once again when the event leaves the window.

On Thu, Apr 21, 2016 at 10:14 PM, John Sherwood <[hidden email]> wrote:

You are looking for sliding windows: https://flink.apache.org/news/2015/12/04/Introducing-windows.html

Here you would do

.timeWindow(Time.seconds(5), Time.seconds(1))

On Thu, Apr 21, 2016 at 12:06 PM, Jonathan Yom-Tov <[hidden email]> wrote:

hi,

Is it possible to implement a continuous time window with flink? Here's an
example. Say I want to count events within a window. The window length is 5
seconds and I get events at t = 1, 2, 7, 8 seconds. I would then expect to
get events with a count at t = 1 (count = 1), t = 2 (count = 2), t = 6
(count = 1), t = 7 (count = 2), t = 8 (count = 2), t = 12 (count = 1) and t
= 13 (count = 0).

How would I go about doing that?.

thanks,
Jon.

[Jonathan Yom-Tov]

Thanks for taking the time. That seems like it would complicated without good knowledge of the overall architecture. I might give it a shot anyway.

On Fri, Apr 22, 2016 at 4:22 PM, Fabian Hueske <[hidden email]> wrote:

Hi Jonathan,

I thought about your use case again. I'm afraid, the approach I proposed is not working due to limitations of the Evictor interface.

The only way that I see to implement you use case is to implement a custom stream operator by extending AbstractStreamOperator and implementing the OneInputStreamOperator interface.

The operator is called for each arriving element and offers timed call-backs. You would have to take care of buffering the elements, registering timers, and emitting elements yourself.

If you do that, you should make sure that all state is kept in Flink's managed state to make sure that your operator can recover from failures.

Cheers, Fabian

2016-04-21 23:16 GMT+02:00 Jonathan Yom-Tov <[hidden email]>:

Thanks. Any pointers on how to do that? Or code examples which do similar things?

On Thu, Apr 21, 2016 at 10:30 PM, Fabian Hueske <[hidden email]> wrote:

Yes, sliding windows are different.

You want to evaluate the window whenever a new element arrives or an element leaves because 5 secs passed since it entered the window, right?

I think that should be possible with a GlobalWindow, a custom Trigger which holds state about the time when each element in the window entered the window, and an Evictor.

2016-04-21 21:19 GMT+02:00 Jonathan Yom-Tov <[hidden email]>:

I think sliding windows are different. In the example in the blog post a window is computed every 30 seconds (so at fixed time intervals). What I want is for a window to be computed every time an event comes in and then once again when the event leaves the window.

On Thu, Apr 21, 2016 at 10:14 PM, John Sherwood <[hidden email]> wrote:

You are looking for sliding windows: https://flink.apache.org/news/2015/12/04/Introducing-windows.html

Here you would do

.timeWindow(Time.seconds(5), Time.seconds(1))

On Thu, Apr 21, 2016 at 12:06 PM, Jonathan Yom-Tov <[hidden email]> wrote:

hi,

Is it possible to implement a continuous time window with flink? Here's an
example. Say I want to count events within a window. The window length is 5
seconds and I get events at t = 1, 2, 7, 8 seconds. I would then expect to
get events with a count at t = 1 (count = 1), t = 2 (count = 2), t = 6
(count = 1), t = 7 (count = 2), t = 8 (count = 2), t = 12 (count = 1) and t
= 13 (count = 0).

How would I go about doing that?.

thanks,
Jon.