A problem of open in AggregateFunction

[Guobao Li]

Hi community,

 

I am trying to register a metric in an aggregate UDF by overriding the open function. According to the documentation, the open function can be override in order to retrieve the metric group to do the metric registration. But it works only on ScalarFunction not on AggregateFunction. Since the open function is not invoked by AggregateFunction. Could anyone help me out of it?

 

Thanks,

Guobao

 

[Biao Liu]

Hi Guobao,

Are you using table API? I'm not familiar with table API, but for data stream API, generally speaking user could do some initialization through "open" method of "Rich" function, like "RichAggregateFunction".

On Tue, 3 Dec 2019 at 22:44, Guobao Li <[hidden email]> wrote:

Hi community,

 

I am trying to register a metric in an aggregate UDF by overriding the open function. According to the documentation, the open function can be override in order to retrieve the metric group to do the metric registration. But it works only on ScalarFunction not on AggregateFunction. Since the open function is not invoked by AggregateFunction. Could anyone help me out of it?

 

Thanks,

Guobao

 

[Jingsong Li]

Hi Guobao,

Looks like this is from table/SQL API.

You can override public void open(FunctionContext context)

It should work, can you provide more information? Like:

- version

- which planner

- what problem, open method never being invoked?

Best,

Jingsong Lee

On Wed, Dec 4, 2019 at 11:09 AM Biao Liu <[hidden email]> wrote:

Hi Guobao,

Are you using table API? I'm not familiar with table API, but for data stream API, generally speaking user could do some initialization through "open" method of "Rich" function, like "RichAggregateFunction".

Thanks,

Biao /'bɪ.aʊ/

On Tue, 3 Dec 2019 at 22:44, Guobao Li <[hidden email]> wrote:

Hi community,

 

I am trying to register a metric in an aggregate UDF by overriding the open function. According to the documentation, the open function can be override in order to retrieve the metric group to do the metric registration. But it works only on ScalarFunction not on AggregateFunction. Since the open function is not invoked by AggregateFunction. Could anyone help me out of it?

 

Thanks,

Guobao

 

--

[Guobao Li]

Hi,

 

Thanks for the quick response. In fact, I am using a branch which forked the version 1.6. I implemented an AggregateFunction UDF which is used in Flink SQL. The idea is to do groupBy with this UDF and add a metric inside the UDF to track the size of aggregated array. I have override the open method and it seemed it was not called. But I tried with the latest version it worked. I am wondering if there was a problem in 1.6 and it has been fixed in the following releases.

 

Regards,

Guobao

 

From: Jingsong Li <[hidden email]>
Date: Wednesday 4 December 2019 at 04:23
To: Biao Liu <[hidden email]>
Cc: Guobao Li <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: A problem of open in AggregateFunction

 

Hi Guobao,

 

Looks like this is from table/SQL API.

You can override public void open(FunctionContext context)

It should work, can you provide more information? Like:

- version

- which planner

- what problem, open method never being invoked?

 

Best,

Jingsong Lee

 

On Wed, Dec 4, 2019 at 11:09 AM Biao Liu <[hidden email]> wrote:

Hi Guobao,

 

Are you using table API? I'm not familiar with table API, but for data stream API, generally speaking user could do some initialization through "open" method of "Rich" function, like "RichAggregateFunction".

 

Thanks,

Biao /'bɪ.aʊ/

 

 

 

On Tue, 3 Dec 2019 at 22:44, Guobao Li <[hidden email]> wrote:

Hi community,

 

I am trying to register a metric in an aggregate UDF by overriding the open function. According to the documentation, the open function can be override in order to retrieve the metric group to do the metric registration. But it works only on ScalarFunction not on AggregateFunction. Since the open function is not invoked by AggregateFunction. Could anyone help me out of it?

 

Thanks,

Guobao

 

 

--

Best, Jingsong Lee

[Biao Liu]

Hi Guobao,

I just re-checked the data stream API. There is an interesting restriction of `AggregateFunction`. It could not be a rich function. And there are already several relevant issues [1][2][3].

I guess your scenario might be relevant to this restriction (I assume you are using table/SQL API).

The bad news is none of these issues are implemented. I didn't find a workaround way to access metric group in data stream API. I'm not sure it could be done in table/SQL or not.

[1] https://issues.apache.org/jira/browse/FLINK-14275

[2] https://issues.apache.org/jira/browse/FLINK-5094

[3] https://issues.apache.org/jira/browse/FLINK-11198

On Wed, 4 Dec 2019 at 11:23, Jingsong Li <[hidden email]> wrote:

Hi Guobao,

Looks like this is from table/SQL API.

You can override public void open(FunctionContext context)

It should work, can you provide more information? Like:

- version

- which planner

- what problem, open method never being invoked?

Best,

Jingsong Lee

On Wed, Dec 4, 2019 at 11:09 AM Biao Liu <[hidden email]> wrote:

Hi Guobao,

Are you using table API? I'm not familiar with table API, but for data stream API, generally speaking user could do some initialization through "open" method of "Rich" function, like "RichAggregateFunction".

Thanks,

Biao /'bɪ.aʊ/

On Tue, 3 Dec 2019 at 22:44, Guobao Li <[hidden email]> wrote:

Hi community,

 

I am trying to register a metric in an aggregate UDF by overriding the open function. According to the documentation, the open function can be override in order to retrieve the metric group to do the metric registration. But it works only on ScalarFunction not on AggregateFunction. Since the open function is not invoked by AggregateFunction. Could anyone help me out of it?

 

Thanks,

Guobao

 

--

Best, Jingsong Lee