Window data retention - Appending to previous data and processing

[G.S.Vijay Raajaa]

Hi ,

I am trying to implement a flink job which requires a window that keeps on adding data to the previous data in the window. The idea is for every addition of a new stream of record, the subsequent chain till the sink needs to be called. In the next iteration window will have old data + new data and the chain is still processed.

Iteration 1 - Window ( record_1) -> Proceed with downstream Chaining and call sink

Iteration 2 - Window(record_1,record_2) -> Proceed with downstream Chaining and call sink

Iteration n - Window(record_1,record_2,...,record n) -> Proceed with downstream Chaining and call sink

Finally , clear the window at the configured time of the day.

I hope the use case is clear. Looking forward to your thoughts on designing the same.

Regards,

Vijay Raajaa G S

[Aljoscha Krettek]

Hi,

I think you should be able to do this by using:

 * GlobalWindows as your window function
 * a custom Trigger that fires on every element, sets a timer for your cleanup time, and purges when the cleanup timer fires
 * a ProcessWindowFunction, to so that you always get all the contents of the window when processing a window

Best,
Aljoscha

> On 24. Jun 2017, at 18:37, G.S.Vijay Raajaa <[hidden email]> wrote:
>
> Hi ,
>
> I am trying to implement a flink job which requires a window that keeps on adding data to the previous data in the window. The idea is for every addition of a new stream of record, the subsequent chain till the sink needs to be called. In the next iteration window will have old data + new data and the chain is still processed.
>
> Iteration 1 - Window ( record_1) -> Proceed with downstream Chaining and call sink
> Iteration 2 - Window(record_1,record_2) -> Proceed with downstream Chaining and call sink
> Iteration n - Window(record_1,record_2,...,record n) -> Proceed with downstream Chaining and call sink
>
> Finally , clear the window at the configured time of the day.
>
> I hope the use case is clear. Looking forward to your thoughts on designing the same.
>
> Regards,
> Vijay Raajaa G S

[G.S.Vijay Raajaa]

Thanks a lot. It works fine !!

Regards,

Vijay Raajaa GS 

On Mon, Jun 26, 2017 at 7:01 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

I think you should be able to do this by using:

 * GlobalWindows as your window function
 * a custom Trigger that fires on every element, sets a timer for your cleanup time, and purges when the cleanup timer fires
 * a ProcessWindowFunction, to so that you always get all the contents of the window when processing a window

Best,
Aljoscha

> On 24. Jun 2017, at 18:37, G.S.Vijay Raajaa <[hidden email]> wrote:
>
> Hi ,
>
> I am trying to implement a flink job which requires a window that keeps on adding data to the previous data in the window. The idea is for every addition of a new stream of record, the subsequent chain till the sink needs to be called. In the next iteration window will have old data + new data and the chain is still processed.
>
> Iteration 1 - Window ( record_1) -> Proceed with downstream Chaining and call sink
> Iteration 2 - Window(record_1,record_2) -> Proceed with downstream Chaining and call sink
> Iteration n - Window(record_1,record_2,...,record n) -> Proceed with downstream Chaining and call sink
>
> Finally , clear the window at the configured time of the day.
>
> I hope the use case is clear. Looking forward to your thoughts on designing the same.
>
> Regards,
> Vijay Raajaa G S