Size of a window without explicit trigger/evictor

[nsengupta]

Hello Fabian <[hidden email]>

A small question: during the course of our recent conversation on the behaviour of window,trigger and evictor, you had mentioned that if I - the application programmer - do not attach a trigger to a window, Flink will attach one by itself. This trigger ensures that the size of the window never grows beyond a threshold, thereby ensuring that a burgeoning window never inflicts a OOM on Flink.

Is this a special Trigger? What's the name of the class? Moreover, how is that threshold size (of the window) determined? Is it configurable?

TIA.

-- Nirmalya

--

[Fabian Hueske-2]

Hi Nirmalya,

sorry for the delayed answer.

First of all, Flink does not take care that our windows fit into memory.
The default trigger depends on the way in which you define a window. Given a KeyedStream you can define a window in the following ways:

KeyedStream s = ...

s.timeWindow() // this will use an EventTimeTrigger or ProcessingTimeTrigger, depending on the time characteristics of the stream

s.countWindow() // this will use a CountTrigger

s.window(? extends WindowAssigner) // this will use the default trigger as defined by the WindowAssigner

None of these triggers monitors the JVM heap to prevent OOMs. If you define a TimeTrigger for one hour and receive too much data, the program will crash. IMO, this behavior is preferable over early triggering which would cause semantically wrong results. If you use a ReduceFunction to compute the result of a window (and no Evictor), the window result can be partially aggregated and its state does not grow.

Best, Fabian

2015-12-10 2:47 GMT+01:00 Nirmalya Sengupta <[hidden email]>:

Hello Fabian <[hidden email]>

A small question: during the course of our recent conversation on the behaviour of window,trigger and evictor, you had mentioned that if I - the application programmer - do not attach a trigger to a window, Flink will attach one by itself. This trigger ensures that the size of the window never grows beyond a threshold, thereby ensuring that a burgeoning window never inflicts a OOM on Flink.

Is this a special Trigger? What's the name of the class? Moreover, how is that threshold size (of the window) determined? Is it configurable?

TIA.

-- Nirmalya

--

Software Technologist
http://www.linkedin.com/in/nirmalyasengupta
"If you have built castles in the air, your work need not be lost. That is where they should be.
Now put the foundation under them."