Unexpected key-group in restore

[Alexey Trenikhun]

Hello,

Suddenly our job (Flink 1.12.2) started to failing to restore from savepoint due to unexpected key group, it looks like stack is for memory state backend, while in fact actually in flink-conf.yaml state.backend: rocksdb

{"ts":"2021-05-11T15:44:09.004Z","message":"Loading configuration property: state.backend, rocksdb","logger_name":"org.apache.flink.configuration.GlobalConfiguration","thread_name":"main","level":"INFO","level_value":20000}

...

{"ts":"2021-05-11T15:45:29.440Z","message":"Xform Voice Ux (1/3) (8b539f9d1794cd3bcedecaa2082cecf2) switched from RUNNING to FAILED on 10.204.2.162:6122-f7a908 @ gsp-tm-0.gsp-headless.gsp.svc.cluster.local (dataPort=37741).","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-2","level":"INFO","level_value":20000,"stack_trace":"java.lang.Exception: Exception while creating StreamOperatorStateContext.\n\tat org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)\n\tat org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)\n\tat org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:427)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:543)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:533)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)\n\tat org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)\n\tat org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for KeyedProcessOperator_6c9209d0aee06b8c3f8bb113696c4bb2_(1/3) from any of the 1 provided restore options.\n\tat org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)\n\tat org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)\n\tat org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)\n\t... 9 common frames omitted\nCaused by: org.apache.flink.runtime.state.BackendBuildingException: Failed when trying to restore heap backend\n\tat org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:115)\n\tat org.apache.flink.runtime.state.memory.MemoryStateBackend.createKeyedStateBackend(MemoryStateBackend.java:357)\n\tat org.apache.flink.runtime.state.memory.MemoryStateBackend.createKeyedStateBackend(MemoryStateBackend.java:104)\n\tat org.apache.flink.runtime.state.StateBackend.createKeyedStateBackend(StateBackend.java:181)\n\tat org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)\n\tat org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)\n\tat org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)\n\t... 11 common frames omitted\nCaused by: java.lang.IllegalStateException: Unexpected key-group in restore.\n\tat org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)\n\tat org.apache.flink.runtime.state.heap.HeapRestoreOperation.readStateHandleStateData(HeapRestoreOperation.java:279)\n\tat org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:172)\n\tat org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:112)\n\t... 17 common frames omitted\n"}

Thanks,

Alexey