Firing windows multiple times

[Aljoscha Krettek]

Hi,

I'd be very happy to give you pointers for FLIP-2 and FLIP-4. Why don't you start a separate thread on the dev list so that we don't hijack this thread.

For FLIP-4 we also have to coordinate with Vishnu, he was driving FLIP-4 but lately everyone has been a bit inactive on that. Let's see if he as anything to say, I'll loop him in directly.

Cheers,

Aljoscha

On Thu, 8 Sep 2016 at 21:48 aj.h <[hidden email]> wrote:

Hi, I'm interested in helping out on this project. I also want to implement a
continuous time-boxed sliding window, my current use case is a 60-second
sliding window that moves whenever a newer event arrives, discarding any
late events that arrive outside the current window, but *also* re-triggering
window processing for any late events within the current window. I
considered using sliding windows with a 1-second granularity, but I'd be
discarding a lot of windows on sparse data, and rebuilding pontetially very
large windows for relatively small 1-second updates.

I'm a fellow in the Insight Data Engineering program. We just got underway,
and I have 3 weeks in which to complete a project. I'd love to tackle this
one, and I'm trying to assess the practicality and feasibility of it.

I noticed that FLIP-2 and FLIP-4 are still under discussion; is it premature
to try to implement these enhancements? And would you be at all
willing/available to help me get up to speed?

Thank you much!



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Firing-windows-multiple-times-tp8424p8975.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[vishnuviswanath]

Hi,

Sorry, I was little busy lately. 

I have most part of FLIP-4 ready except ProcessingTimeEvictor case. I was trying to find the best way to set the ProcessingTime as per the suggestion from Aljoscha then I got busy with some other work. 

We can definitely work together on that.

Thanks

Vishnu

On Fri, Sep 9, 2016 at 3:29 AM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

I'd be very happy to give you pointers for FLIP-2 and FLIP-4. Why don't you start a separate thread on the dev list so that we don't hijack this thread.

For FLIP-4 we also have to coordinate with Vishnu, he was driving FLIP-4 but lately everyone has been a bit inactive on that. Let's see if he as anything to say, I'll loop him in directly.

Cheers,

Aljoscha

On Thu, 8 Sep 2016 at 21:48 aj.h <[hidden email]> wrote:

Hi, I'm interested in helping out on this project. I also want to implement a
continuous time-boxed sliding window, my current use case is a 60-second
sliding window that moves whenever a newer event arrives, discarding any
late events that arrive outside the current window, but *also* re-triggering
window processing for any late events within the current window. I
considered using sliding windows with a 1-second granularity, but I'd be
discarding a lot of windows on sparse data, and rebuilding pontetially very
large windows for relatively small 1-second updates.

I'm a fellow in the Insight Data Engineering program. We just got underway,
and I have 3 weeks in which to complete a project. I'd love to tackle this
one, and I'm trying to assess the practicality and feasibility of it.

I noticed that FLIP-2 and FLIP-4 are still under discussion; is it premature
to try to implement these enhancements? And would you be at all
willing/available to help me get up to speed?

Thank you much!



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Firing-windows-multiple-times-tp8424p8975.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[aj.h]

In the way that FLIP-2 would solve this problem, secondAggregate would ignore the early firing updates from firstAggregate to prevent double-counting, correct? If that's the case, I am trying to understand why we'd want to trigger early-fires every 30 seconds for the secondAggregate if it's only accepting new results at a daily rate, after firstAggregate's primary firing at the end of the window. If we filter out results from early-fires, wouldn't every 30-second result from secondAggregate remain unchanged within the same 1-day window?

Similarly (compounded) for a 365-day window aggregating over a 30 day window: if it filters out early fires, wouldn't it only produce new/unique results every 30 days?

I very well may have misunderstood this solution.

[Aljoscha Krettek]

Hi,

yes AJ that observation is correct. Let's see what Shannon has to say about this but it might be that all "higher-level" aggregates will have to be based on the first level and can then update at the speed of that aggregate.

Cheers,

Aljoscha

On Mon, 12 Sep 2016 at 05:03 aj.h <[hidden email]> wrote:

In the way that FLIP-2 would solve this problem, secondAggregate would ignore
the early firing updates from firstAggregate to prevent double-counting,
correct? If that's the case, I am trying to understand why we'd want to
trigger early-fires every 30 seconds for the secondAggregate if it's only
accepting new results at a daily rate, after firstAggregate's primary firing
at the end of the window. If we filter out results from early-fires,
wouldn't every 30-second result from secondAggregate remain unchanged within
the same 1-day window?

Similarly (compounded) for a 365-day window aggregating over a 30 day
window: if it filters out early fires, wouldn't it only produce new/unique
results every 30 days?

I very well may have misunderstood this solution.



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Firing-windows-multiple-times-tp8424p8994.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.