Re: Size of a window without explicit trigger/evictor

Posted by Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Size-of-a-window-without-explicit-trigger-evictor-tp3999p4087.html

Hi Nirmalya,

sorry for the delayed answer.

First of all, Flink does not take care that our windows fit into memory.
The default trigger depends on the way in which you define a window. Given a KeyedStream you can define a window in the following ways:

KeyedStream s = ...

s.timeWindow() // this will use an EventTimeTrigger or ProcessingTimeTrigger, depending on the time characteristics of the stream

s.countWindow() // this will use a CountTrigger

s.window(? extends WindowAssigner) // this will use the default trigger as defined by the WindowAssigner

None of these triggers monitors the JVM heap to prevent OOMs. If you define a TimeTrigger for one hour and receive too much data, the program will crash. IMO, this behavior is preferable over early triggering which would cause semantically wrong results. If you use a ReduceFunction to compute the result of a window (and no Evictor), the window result can be partially aggregated and its state does not grow.

Best, Fabian

2015-12-10 2:47 GMT+01:00 Nirmalya Sengupta <[hidden email]>:

Hello Fabian <[hidden email]>

A small question: during the course of our recent conversation on the behaviour of window,trigger and evictor, you had mentioned that if I - the application programmer - do not attach a trigger to a window, Flink will attach one by itself. This trigger ensures that the size of the window never grows beyond a threshold, thereby ensuring that a burgeoning window never inflicts a OOM on Flink.

Is this a special Trigger? What's the name of the class? Moreover, how is that threshold size (of the window) determined? Is it configurable?

TIA.

-- Nirmalya

--

Software Technologist
http://www.linkedin.com/in/nirmalyasengupta
"If you have built castles in the air, your work need not be lost. That is where they should be.
Now put the foundation under them."