Propagating delta from window upon trigger

[Nikhil Goyal]

Hi guys,

Is there a way in Flink to only propagate the changes which happened in the window's state rather than dumbing the contents of the window again and again upon trigger?

Thanks
Nikhil

[miki haiat]

Can you elaborate more what  is you use case ?

On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <[hidden email]> wrote:

Hi guys,

Is there a way in Flink to only propagate the changes which happened in the window's state rather than dumbing the contents of the window again and again upon trigger?

Thanks
Nikhil

[Nikhil Goyal]

I have window of 1 hour and trigger of 5 min. I want to know if every 5 min Flink is writing the entire window or only the keys which changed.

On Sat, May 18, 2019, 9:40 AM miki haiat <[hidden email]> wrote:

Can you elaborate more what  is you use case ?

On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <[hidden email]> wrote:

Hi guys,

Is there a way in Flink to only propagate the changes which happened in the window's state rather than dumbing the contents of the window again and again upon trigger?

Thanks
Nikhil

[Congxian Qiu]

Hi, Nikhil

Window will emit all state to downstream. Can you clear the state while triggering? 

Nikhil Goyal <[hidden email]>于2019年5月19日 周日01:03写道：

I have window of 1 hour and trigger of 5 min. I want to know if every 5 min Flink is writing the entire window or only the keys which changed.

On Sat, May 18, 2019, 9:40 AM miki haiat <[hidden email]> wrote:

Can you elaborate more what  is you use case ?

On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <[hidden email]> wrote:

Hi guys,

Is there a way in Flink to only propagate the changes which happened in the window's state rather than dumbing the contents of the window again and again upon trigger?

Thanks
Nikhil

--

[Rong Rong]

Hi Miki,

Upon trigger, window will be fired with all of its content in its state invoking the "emitWindowContent" method. which will further invoke the window function you define. 

Thus if your goal is to only emit the delta, one thing is to do so in your window function. One challenge you might face is how to make your window function stateful, you might want to checkout this document[1] for more details.

If your goal is to limit the number of content stored in the window state to only the delta. you can also look at incremental aggregation state [2] of the window, but you will have to design your window function in such a way that it can take delta into account. 

--

Rong

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction

[2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction-with-incremental-aggregation

On Sun, May 19, 2019 at 8:00 PM Congxian Qiu <[hidden email]> wrote:

Hi, Nikhil

Window will emit all state to downstream. Can you clear the state while triggering? 

Nikhil Goyal <[hidden email]>于2019年5月19日 周日01:03写道：

I have window of 1 hour and trigger of 5 min. I want to know if every 5 min Flink is writing the entire window or only the keys which changed.

On Sat, May 18, 2019, 9:40 AM miki haiat <[hidden email]> wrote:

Can you elaborate more what  is you use case ?

On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <[hidden email]> wrote:

Hi guys,

Is there a way in Flink to only propagate the changes which happened in the window's state rather than dumbing the contents of the window again and again upon trigger?

Thanks
Nikhil

--

[Nikhil Goyal]

Thanks Rong, I will look at the links you mentioned.

On Wed, May 22, 2019 at 10:39 AM Rong Rong <[hidden email]> wrote:

Hi Miki,

Upon trigger, window will be fired with all of its content in its state invoking the "emitWindowContent" method. which will further invoke the window function you define. 

Thus if your goal is to only emit the delta, one thing is to do so in your window function. One challenge you might face is how to make your window function stateful, you might want to checkout this document[1] for more details.

If your goal is to limit the number of content stored in the window state to only the delta. you can also look at incremental aggregation state [2] of the window, but you will have to design your window function in such a way that it can take delta into account. 

--

Rong

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction

[2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#processwindowfunction-with-incremental-aggregation

On Sun, May 19, 2019 at 8:00 PM Congxian Qiu <[hidden email]> wrote:

Hi, Nikhil

Window will emit all state to downstream. Can you clear the state while triggering? 

Nikhil Goyal <[hidden email]>于2019年5月19日 周日01:03写道：

I have window of 1 hour and trigger of 5 min. I want to know if every 5 min Flink is writing the entire window or only the keys which changed.

On Sat, May 18, 2019, 9:40 AM miki haiat <[hidden email]> wrote:

Can you elaborate more what  is you use case ?

On Sat, May 18, 2019 at 12:47 AM Nikhil Goyal <[hidden email]> wrote:

Hi guys,

Is there a way in Flink to only propagate the changes which happened in the window's state rather than dumbing the contents of the window again and again upon trigger?

Thanks
Nikhil

--