Avoid re-counting overlapping elements

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Avoid re-counting overlapping elements

hassahma

 i am implementing the following workflow using SlidingProcessingTimeWindows:

events.keyBy("key")
    .window(SlidingProcessingTimeWindows.of(Time.seconds(10),Time.seconds(5)))
    .fold(new Tuple3<List, Integer, Integer>(new ArrayList<String>(), 0, 0), new FoldMapper(), new MyCountWindowFunction())
    .print()

Lets say events are coming into window A and B:

Window A: 1 2 3 4 5
Window B:       3 4 5 1 2 3 4 5

I have two questions:

1) How can avoid recounting the elements that I have already counted in the prior window overlapping area (overlapped 5 seconds). i just want to reuse the count of previous window for the overlapped time (last 5 seconds of each window) and then in the new window I want to skip counting the overlapped area and add the result that I received from the previous window. In the above scenario, I would be counting 3,4,5 twice, once for window A and once for window B. I want to do that calculation only in window A and reuse the result in window B. Is it possible in flink?

Thanks!

Regards,
Hassan