Updating stored window data

[Paul Joireman]

Hello,

I'd like to specify a long time duration window (say 1 day) but write a custom trigger to force processing on a more frequent interval (say 10 minutes).  Similar, at least initially, to the thread recently started by S. Carey (Firing windows mulitple times).   However, each time the window function gets the data to process is it possible to update the data objects received by the iterator and have those stored back in the "data buffer" maintained by the window.     Or would this situation be better handled by using a ListState in parallel to the window functionality?  The ListState storing the updated data.

Regards,

Paul

[Ufuk Celebi]

Hey Paul! I think the window content should not be updated. Still,
from looking at some of the internal Flink code, it looks like the
updates would affect the "data buffer" -- but I think that this is
only true for some cases and does not hold in general. Cc'ing Aljoscha
for a definite answer on the subject. In any case, the JavaDocs could
be clearer on this.

On Fri, Aug 12, 2016 at 5:18 PM, Paul Joireman <[hidden email]> wrote:

> Hello,
>
>
> I'd like to specify a long time duration window (say 1 day) but write a
> custom trigger to force processing on a more frequent interval (say 10
> minutes).  Similar, at least initially, to the thread recently started by S.
> Carey (Firing windows mulitple times).   However, each time the window
> function gets the data to process is it possible to update the data objects
> received by the iterator and have those stored back in the "data buffer"
> maintained by the window.     Or would this situation be better handled by
> using a ListState in parallel to the window functionality?  The ListState
> storing the updated data.
>
>
> Regards,
>
> Paul

[Aljoscha Krettek]

Hi,

the input elements to a window function should not be modified. Could you maybe achieve something using a Fold? Maybe if you went into a bit more details we could figure something out together.

Cheers,

Aljoscha

On Tue, 16 Aug 2016 at 10:38 Ufuk Celebi <[hidden email]> wrote:

Hey Paul! I think the window content should not be updated. Still,
from looking at some of the internal Flink code, it looks like the
updates would affect the "data buffer" -- but I think that this is
only true for some cases and does not hold in general. Cc'ing Aljoscha
for a definite answer on the subject. In any case, the JavaDocs could
be clearer on this.

On Fri, Aug 12, 2016 at 5:18 PM, Paul Joireman <[hidden email]> wrote:
> Hello,
>
>
> I'd like to specify a long time duration window (say 1 day) but write a
> custom trigger to force processing on a more frequent interval (say 10
> minutes).  Similar, at least initially, to the thread recently started by S.
> Carey (Firing windows mulitple times).   However, each time the window
> function gets the data to process is it possible to update the data objects
> received by the iterator and have those stored back in the "data buffer"
> maintained by the window.     Or would this situation be better handled by
> using a ListState in parallel to the window functionality?  The ListState
> storing the updated data.
>
>
> Regards,
>
> Paul