Rich and incrementally aggregating window functions

I want to use ProcessWindowFunction.Context#globalState in my window function. But I don't want to apply ProcessWindowFunction directly to my WindowedStream because I don't want to buffer all the elements of each window. Currently I'm using WindowedStream#aggregate(AggregateFunction, ProcessWindowFunction).

My understanding is that RichFunction.runtimeContext also give access to those global states. That thought naturally pointed me to RichAggregateFunction, RichReduceFunction and RichFoldFunction. However, they all cause runtime error like this:
"AggregateFunction can not be a RichFunction. Please use fold(AggregateFunction, WindowFunction) instead."

So how can I use an incrementally aggregating window function and have access to global states at the same time?

[Hequn Cheng]

Hi,

There is a discussion about this before, you can take a look at it[1].

Best, Hequn

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-implementation-of-aggregate-function-using-a-ProcessFunction-td23473.html#a23531

On Thu, May 9, 2019 at 5:14 AM an0 <[hidden email]> wrote:

I want to use ProcessWindowFunction.Context#globalState in my window function. But I don't want to apply ProcessWindowFunction directly to my WindowedStream because I don't want to buffer all the elements of each window. Currently I'm using WindowedStream#aggregate(AggregateFunction, ProcessWindowFunction).

My understanding is that RichFunction.runtimeContext also give access to those global states. That thought naturally pointed me to RichAggregateFunction, RichReduceFunction and RichFoldFunction. However, they all cause runtime error like this:
"AggregateFunction can not be a RichFunction. Please use fold(AggregateFunction, WindowFunction) instead."

So how can I use an incrementally aggregating window function and have access to global states at the same time?

Thanks. I know reimplementing windowing myself will work but that's a very bad last resort.

I believe it is a very reasonable request. But since someone else has already filed a Jira and it was closed as Won't Fix[1], I won't bother refiling it again. I'll try something else first.

[1]
https://issues.apache.org/jira/browse/FLINK-10250

On 2019/05/09 02:07:00, Hequn Cheng <[hidden email]> wrote:

> Hi,
>
> There is a discussion about this before, you can take a look at it[1].
> Best, Hequn
>
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-implementation-of-aggregate-function-using-a-ProcessFunction-td23473.html#a23531
>
> On Thu, May 9, 2019 at 5:14 AM an0 <[hidden email]> wrote:
>
> > I want to use ProcessWindowFunction.Context#globalState in my window
> > function. But I don't want to apply ProcessWindowFunction directly to my
> > WindowedStream because I don't want to buffer all the elements of each
> > window. Currently I'm using WindowedStream#aggregate(AggregateFunction,
> > ProcessWindowFunction).
> >
> > My understanding is that RichFunction.runtimeContext also give access to
> > those global states. That thought naturally pointed me to
> > RichAggregateFunction, RichReduceFunction and RichFoldFunction. However,
> > they all cause runtime error like this:
> > "AggregateFunction can not be a RichFunction. Please use
> > fold(AggregateFunction, WindowFunction) instead."
> >
> > So how can I use an incrementally aggregating window function and have
> > access to global states at the same time?
> >
>