Manually clean SQL keyed state

[shkob1]

I have a scenario in which i do a non-windowed group by using SQL. something
like

"Select count(*) as events, shouldTrigger(..) as shouldTrigger from source
group by sessionId"
i'm then converting to a retracted stream, filtering by "add" messages, then
further filtering by "shouldTrigger" field and sends out the result to a
sink.

While i'm using the query config (idle state retention time), it seems like
i can reduce the state size by clearing the state of the specific session id
earlier ("shouldTrigger" marks the end of the session rather than a timed
window).

Is there a way for me to clear that state assuming i need to use the SQL
API?

Thanks!
Shahar





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Fabian Hueske-2]

Hi Shahar,

That's not possible at the moment. The SQL API does not provide any knobs to control state size besides the idle state retention.

The reason is that it aims to be as accurate as possible.

In the future it might be possible to provide more information to the system (like constraints in relational DBMS) that can help to reduce the memory footprint.

Best, Fabian

Am Fr., 9. Nov. 2018 um 02:06 Uhr schrieb shkob1 <[hidden email]>:

I have a scenario in which i do a non-windowed group by using SQL. something
like

"Select count(*) as events, shouldTrigger(..) as shouldTrigger from source
group by sessionId"
i'm then converting to a retracted stream, filtering by "add" messages, then
further filtering by "shouldTrigger" field and sends out the result to a
sink.

While i'm using the query config (idle state retention time), it seems like
i can reduce the state size by clearing the state of the specific session id
earlier ("shouldTrigger" marks the end of the session rather than a timed
window).

Is there a way for me to clear that state assuming i need to use the SQL
API?

Thanks!
Shahar





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[shkob1]

Thanks Fabian!



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/