StateMigrationException thrown by state processor api

[Paul Lam]

Hi,

I was using the new state processor api to read a savepoint produced by Flink 1.5.3, and got an StateMigrationException with message “For heap backends, the new namespace serializer must be compatible”.

Concretely, the state I was trying to read is a MapState<String, Long> within a WindowOperator(TriggerContext) which is keyed by a string field, so the key & value data types and the corresponding serializers should be the basic one here (POJOSerializer).

However, the error indicates the problem is about namespace serializer which I have not much knowledge about. So I dig a bit into the source code to find that the namespace serializer is not a highly configurable one, and both state processor api and the state to be read seem to use the same namespace serializer, `VoidNamespaceSerializer` (please correct me if I’m wrong). It still doesn’t explain why the error happened. 

Please point me to the right direction. Thanks a lot!

Best,

Paul Lam

[Yun Tang]

Hi Paul

Would you please share more information of the exception stack trace and the state descriptor of this map state with that window operator?

For all user-facing keyed state, the namespace serializer would always be VoidNamespaceSerializer. And only window state could have different name space serializer.

Best

Yun Tang

---

From: Paul Lam <[hidden email]>
Sent: Tuesday, August 27, 2019 17:14
To: user <[hidden email]>
Cc: Tzu-Li (Gordon) Tai <[hidden email]>
Subject: StateMigrationException thrown by state processor api

 

Hi,

I was using the new state processor api to read a savepoint produced by Flink 1.5.3, and got an StateMigrationException with message “For heap backends, the new namespace serializer must be compatible”.

Concretely, the state I was trying to read is a MapState<String, Long> within a WindowOperator(TriggerContext) which is keyed by a string field, so the key & value data types and the corresponding serializers should be the basic one here (POJOSerializer).

However, the error indicates the problem is about namespace serializer which I have not much knowledge about. So I dig a bit into the source code to find that the namespace serializer is not a highly configurable one, and both state processor api and the state to be read seem to use the same namespace serializer, `VoidNamespaceSerializer` (please correct me if I’m wrong). It still doesn’t explain why the error happened. 

Please point me to the right direction. Thanks a lot!

Best,

Paul Lam

[Paul Lam]

Hi Yun,

Thanks to your input, I’ve found out that states in WindowOperator are using its window serializer as the namespace serializer, so the states in TriggerContext can not be deserialized by the state processor API at the moment, as it’s using VoidNamespaceSerializer. Indeed it does ring a bell, I might have read about the limitation that the state processor API is unable to read states in WindowOperator currently. All in all, thanks again.

Best,

Paul Lam

在 2019年8月27日，17:32，Yun Tang <[hidden email]> 写道：

Hi Paul

Would you please share more information of the exception stack trace and the state descriptor of this map state with that window operator?

For all user-facing keyed state, the namespace serializer would always be VoidNamespaceSerializer. And only window state could have different name space serializer.

Best

Yun Tang

---

From: Paul Lam <[hidden email]>
Sent: Tuesday, August 27, 2019 17:14
To: user <[hidden email]>
Cc: Tzu-Li (Gordon) Tai <[hidden email]>
Subject: StateMigrationException thrown by state processor api

 

Hi,

I was using the new state processor api to read a savepoint produced by Flink 1.5.3, and got an StateMigrationException with message “For heap backends, the new namespace serializer must be compatible”.

Concretely, the state I was trying to read is a MapState<String, Long> within a WindowOperator(TriggerContext) which is keyed by a string field, so the key & value data types and the corresponding serializers should be the basic one here (POJOSerializer).

However, the error indicates the problem is about namespace serializer which I have not much knowledge about. So I dig a bit into the source code to find that the namespace serializer is not a highly configurable one, and both state processor api and the state to be read seem to use the same namespace serializer, `VoidNamespaceSerializer` (please correct me if I’m wrong). It still doesn’t explain why the error happened. 

Please point me to the right direction. Thanks a lot!

Best,

Paul Lam