How to debug window states

[Paul Lam]

Hi,

Since currently State Processor doesn’t support window states, what’s the recommended way to debug window states?

The background is that I have a SQL job whose states, mainly window aggregation operator states, keep increasing steadily.

The things I’ve checked:

- Outputs are as expected.

- Keys amount are capped.

- Watermarks are good, no lags.

- No back pressure.

If I can directly read the states to find out what’s accountable for the state size growth, that would be very intuitional and time-saving.

Best,

Paul Lam

[Paul Lam]

It turns out to be a bug of StateTTL [1]. But I’m still interested in debugging window states. 

Any suggestions are appreciated. Thanks!

1. https://issues.apache.org/jira/browse/FLINK-16581

Best,

Paul Lam

2020年7月15日 13:13，Paul Lam <[hidden email]> 写道：

Hi,

Since currently State Processor doesn’t support window states, what’s the recommended way to debug window states?

The background is that I have a SQL job whose states, mainly window aggregation operator states, keep increasing steadily.

The things I’ve checked:

- Outputs are as expected.

- Keys amount are capped.

- Watermarks are good, no lags.

- No back pressure.

If I can directly read the states to find out what’s accountable for the state size growth, that would be very intuitional and time-saving.

Best,

Paul Lam

[Congxian Qiu]

Hi

There is an issue[1] wants to provide an easy way to read/bootstrap window state using the state processor API, but the PR was closed currently. 

[1] https://issues.apache.org/jira/browse/FLINK-13095

Paul Lam <[hidden email]> 于2020年7月15日周三 下午4:21写道：

It turns out to be a bug of StateTTL [1]. But I’m still interested in debugging window states. 

Any suggestions are appreciated. Thanks!

1. https://issues.apache.org/jira/browse/FLINK-16581

Best,

Paul Lam

2020年7月15日 13:13，Paul Lam <[hidden email]> 写道：

Hi,

Since currently State Processor doesn’t support window states, what’s the recommended way to debug window states?

The background is that I have a SQL job whose states, mainly window aggregation operator states, keep increasing steadily.

The things I’ve checked:

- Outputs are as expected.

- Keys amount are capped.

- Watermarks are good, no lags.

- No back pressure.

If I can directly read the states to find out what’s accountable for the state size growth, that would be very intuitional and time-saving.

Best,

Paul Lam