Regarding caching the evicted elements and re-emitting them to the next window

[Abdul Salam Shaikh]

Hi,

I am using 1.2-Snapshot version of Apache Flink which provides the new enhanced Evictor functionality and using customized triggers for Global Window. I have a use case where I am evicting the unwanted event(element) for the current window before it is evaluated. However, I am looking for options to cache this evicted element and re-use it in the next window. Is there a possibility which can help me achieve this in the context of Flink or in a more generic programming approach.

Thanks in anticipation! 

[Aljoscha Krettek]

Hi,
I'm afraid there is no functionality for this in Flink. What you can do, however, is to not evict these elements from the window buffer but instead ignore them when processing your elements in the WindowFunction. This way they will be preserved for the next firing. You have to make sure to eventually evict some elements, however. Otherwise you would have a memory leak.

Aljoscha

On Sun, Jan 8, 2017, 23:47 Abdul Salam Shaikh <[hidden email]> wrote:

Hi,

I am using 1.2-Snapshot version of Apache Flink which provides the new enhanced Evictor functionality and using customized triggers for Global Window. I have a use case where I am evicting the unwanted event(element) for the current window before it is evaluated. However, I am looking for options to cache this evicted element and re-use it in the next window. Is there a possibility which can help me achieve this in the context of Flink or in a more generic programming approach.

Thanks in anticipation! 

[Shaoxuan Wang]

Hi Abdul,

You may want to check out FLIP13 "side output" https://goo.gl/6KSYd0 . Once we have this feature, you should be able to collect the data to the external distributed storage, and use these data later on demand. 

BTW, can you explain your use case in more details, such that people here may help you figure out a better solution (it perhaps may just need some tunings on your query plan).

Shaoxuan

On Sat, Jan 14, 2017 at 12:22 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,
I'm afraid there is no functionality for this in Flink. What you can do, however, is to not evict these elements from the window buffer but instead ignore them when processing your elements in the WindowFunction. This way they will be preserved for the next firing. You have to make sure to eventually evict some elements, however. Otherwise you would have a memory leak.

Aljoscha

On Sun, Jan 8, 2017, 23:47 Abdul Salam Shaikh <[hidden email]> wrote:

Hi,

I am using 1.2-Snapshot version of Apache Flink which provides the new enhanced Evictor functionality and using customized triggers for Global Window. I have a use case where I am evicting the unwanted event(element) for the current window before it is evaluated. However, I am looking for options to cache this evicted element and re-use it in the next window. Is there a possibility which can help me achieve this in the context of Flink or in a more generic programming approach.

Thanks in anticipation!