Cannot access state from a empty taskmanager - using kubernetes

[Daniel Peled]

Hi, 

We have followed the instructions in the following link ""Enabling Queryable State" with kubernetes:

https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/standalone/kubernetes.html#enabling-queryable-state

When the replicas of the task-manager pods is 1 we get NO error
But when the replicas is greater than 1 for example 7 we get the following error when trying to access flink state:
We think it might be related to jira issue FLINK-10225 that has been abandoned 

java.util.concurrent.ExecutionException: java.lang.RuntimeException: Failed request 1.
 Caused by: org.apache.flink.queryablestate.exceptions.UnknownLocationException: Could not retrieve location of state=stopJobValueState of job=d5d14923157f5c3d3c4b2e1b7c02a942. Potential reasons are: i) the state is not ready, or ii) the job does not exist.
 Caused by: org.apache.flink.queryablestate.exceptions.UnknownLocationException: Could not retrieve location of state=stopJobValueState of job=d5d14923157f5c3d3c4b2e1b7c02a942. Potential reasons are: i) the state is not ready, or ii) the job does not exist.

at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.getKvStateLookupInfo(KvStateClientProxyHandler.java:247)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.getState(KvStateClientProxyHandler.java:164)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.executeActionAsync(KvStateClientProxyHandler.java:131)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.handleRequest(KvStateClientProxyHandler.java:121)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.handleRequest(KvStateClientProxyHandler.java:63)
at org.apache.flink.queryablestate.network.AbstractServerHandler$AsyncRequestTask.run(AbstractServerHandler.java:258)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

BR,

Danny

[Chesnay Schepler]

Yes, it does sound quite a lot like FLINK-10225. I assume it is only happening for some task executors and not all of them?

Unfortunately I don't think this issue will be fixed anytime soon.

On 1/28/2021 12:59 PM, Daniel Peled wrote:

Hi, 

We have followed the instructions in the following link ""Enabling Queryable State" with kubernetes:

https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/standalone/kubernetes.html#enabling-queryable-state

When the replicas of the task-manager pods is 1 we get NO error
But when the replicas is greater than 1 for example 7 we get the following error when trying to access flink state:
We think it might be related to jira issue FLINK-10225 that has been abandoned 

java.util.concurrent.ExecutionException: java.lang.RuntimeException: Failed request 1.
 Caused by: org.apache.flink.queryablestate.exceptions.UnknownLocationException: Could not retrieve location of state=stopJobValueState of job=d5d14923157f5c3d3c4b2e1b7c02a942. Potential reasons are: i) the state is not ready, or ii) the job does not exist.
 Caused by: org.apache.flink.queryablestate.exceptions.UnknownLocationException: Could not retrieve location of state=stopJobValueState of job=d5d14923157f5c3d3c4b2e1b7c02a942. Potential reasons are: i) the state is not ready, or ii) the job does not exist.

at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.getKvStateLookupInfo(KvStateClientProxyHandler.java:247)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.getState(KvStateClientProxyHandler.java:164)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.executeActionAsync(KvStateClientProxyHandler.java:131)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.handleRequest(KvStateClientProxyHandler.java:121)
at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.handleRequest(KvStateClientProxyHandler.java:63)
at org.apache.flink.queryablestate.network.AbstractServerHandler$AsyncRequestTask.run(AbstractServerHandler.java:258)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

BR,

Danny