org.apache.flink.runtime.rpc.e
Locked
 
|
I deploy flink1.12.2 in standalone mode on k8s whith HA enabled . When I restart the jobmanager pod expecting all pre-submitted jobs to recover ,however, This not happen everytime but quit often. Full log is as follows: JM_RESOURCE_PARAMS extraction logs: jvm_params: -Xmx15703474176 -Xms15703474176 -XX:MaxMetaspaceSize=268435456 logs: INFO [] - Loading configuration property: jobmanager.rpc.address, flink-jobmanager INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 2 INFO [] - Loading configuration property: blob.server.port, 6124 INFO [] - Loading configuration property: jobmanager.rpc.port, 6123 INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 INFO [] - Loading configuration property: queryable-state.proxy.ports, 6125 INFO [] - Loading configuration property: jobmanager.memory.process.size, 16384m INFO [] - Loading configuration property: taskmanager.memory.process.size, 16384m INFO [] - Loading configuration property: parallelism.default, 2 INFO [] - Loading configuration property: metrics.reporters, prom INFO [] - Loading configuration property: metrics.reporter.prom.class, org.apache.flink.metrics.prometheus.PrometheusReporter INFO [] - Loading configuration property: metrics.reporter.prom.port, 9999 INFO [] - Loading configuration property: kubernetes.cluster-id, yewu1122 INFO [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory INFO [] - Loading configuration property: high-availability.storageDir, hdfs:///flink/flink1122-yewu-ha0511 INFO [] - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///flink/flink-historyserver INFO [] - Loading configuration property: historyserver.archive.fs.dir, hdfs:///flink/flink-historyserver INFO [] - Loading configuration property: historyserver.archive.fs.refresh-interval, 100000 INFO [] - Loading configuration property: historyserver.web.port, 8082 INFO [] - Loading configuration property: fs.oss.endpoint, http://*****.com INFO [] - Loading configuration property: fs.oss.accessKeyId, ***** INFO [] - Loading configuration property: fs.oss.accessKeySecret, ****** INFO [] - The derived from fraction jvm overhead memory (1.600gb (1717986944 bytes)) is greater than its max value 1024.000mb (1073741824 bytes), max value will be used instead INFO [] - Final Master Memory configuration: INFO [] - Total Process Memory: 16.000gb (17179869184 bytes) INFO [] - Total Flink Memory: 14.750gb (15837691904 bytes) INFO [] - JVM Heap: 14.625gb (15703474176 bytes) INFO [] - Off-heap: 128.000mb (134217728 bytes) INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) INFO [] - JVM Overhead: 1024.000mb (1073741824 bytes) 17:23,664 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 17:23,664 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneSessionClusterEntrypoint (Version: 1.12.2, Scala: 2.11, Rev:4dedee0, Date:2021-02-26T17:14:28+01:00) 17:23,664 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS current user: flink 17:23,934 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Current Hadoop/Kerberos user: flink 17:23,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.282-b08 17:23,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Maximum heap size: 14476 MiBytes 17:23,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JAVA_HOME: /usr/local/openjdk-8 17:23,936 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Hadoop version: 2.6.5 17:23,936 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM Options: 17:23,936 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xmx15703474176 17:23,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Xms15703474176 17:23,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -XX:MaxMetaspaceSize=268435456 17:23,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog.file=/opt/flink/log/flink--standalonesession-0-jobmanager-5cd8c9dc9-89s67.log 17:23,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties 17:23,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties 17:23,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml 17:23,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Program Arguments: 17:23,938 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --configDir 17:23,938 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - /opt/flink/conf 17:23,938 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --executionMode 17:23,938 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - cluster 17:23,938 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - --host 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - 172.19.89.67 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.off-heap.size=134217728b 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.min=1073741824b 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-metaspace.size=268435456b 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.heap.size=15703474176b 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -D 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - jobmanager.memory.jvm-overhead.max=1073741824b 17:23,939 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Classpath: /opt/flink/lib/flink-csv-1.12.2.jar:/opt/flink/lib/flink-json-1.12.2.jar:/opt/flink/lib/flink-oss-fs-hadoop-1.12.2.jar:/opt/flink/lib/flink-shaded-hadoop-2-uber-2.6.5-10.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-sql-connector-mysql-cdc-1.3.0.jar:/opt/flink/lib/flink-table-blink_2.11-1.12.2.jar:/opt/flink/lib/flink-table_2.11-1.12.2.jar:/opt/flink/lib/iceberg-flink-runtime-myversion.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.11-1.12.2.jar::/etc/hadoop/conf: 17:23,940 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - -------------------------------------------------------------------------------- 17:23,941 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Registered UNIX signal handlers for [TERM, HUP, INT] 17:23,951 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, flink-jobmanager 17:23,951 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 2 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: queryable-state.proxy.ports, 6125 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 16384m 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 16384m 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 2 17:23,952 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporters, prom 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporter.prom.class, org.apache.flink.metrics.prometheus.PrometheusReporter 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: metrics.reporter.prom.port, 9999 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, yewu1122 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability, org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: high-availability.storageDir, hdfs:///flink/flink1122-yewu-ha0511 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.archive.fs.dir, hdfs:///flink/flink-historyserver 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: historyserver.archive.fs.dir, hdfs:///flink/flink-historyserver 17:23,953 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: historyserver.archive.fs.refresh-interval, 100000 17:23,954 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: historyserver.web.port, 8082 17:24,002 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem. 17:24,039 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context. 17:24,081 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to flink (auth:SIMPLE) 17:24,085 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-3626562489480703955.conf. 17:24,094 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services. 17:24,113 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.19.89.67:6123, bind address 0.0.0.0:6123. 17:24,544 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started 17:24,563 INFO akka.remote.Remoting [] - Starting remoting 17:24,670 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink@172.19.89.67:6123] 17:24,726 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink@172.19.89.67:6123 17:25,274 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17:25,290 INFO org.apache.flink.runtime.blob.FileSystemBlobStore [] - Creating highly available BLOB storage directory at hdfs:/flink/flink1122-yewu-ha0511/default/blob 17:25,371 INFO org.apache.flink.runtime.blob.BlobServer [] - Created BLOB server storage directory /tmp/blobStore-8d6de645-fbcb-4c11-a72a-3378b00d51e3 17:25,375 INFO org.apache.flink.runtime.blob.BlobServer [] - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000 17:25,429 INFO org.apache.flink.metrics.prometheus.PrometheusReporter [] - Started PrometheusReporter HTTP server on port 9999. 17:25,429 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - Reporting metrics for reporter prom of type org.apache.flink.metrics.prometheus.PrometheusReporter. 17:25,433 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 172.19.89.67:0, bind address 0.0.0.0:0. 17:25,448 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started 17:25,451 INFO akka.remote.Remoting [] - Starting remoting 17:25,464 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://flink-metrics@172.19.89.67:38693] 17:25,471 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://flink-metrics@172.19.89.67:38693 17:25,485 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService . 17:25,510 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore [] - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-495b19bf-44ce-41c2-9e92-06af8da54f64, expiration time 3600000, maximum cache size 52428800 bytes. 17:25,553 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address' 17:25,770 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component log file: /opt/flink/log/flink--standalonesession-0-jobmanager-5cd8c9dc9-89s67.log 17:25,770 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils [] - Determined location of main cluster component stdout file: /opt/flink/log/flink--standalonesession-0-jobmanager-5cd8c9dc9-89s67.out 17:26,087 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector yewu1122-restserver-leader with lock identity 535eb66e-9240-40dd-90a7-a29ef88b8475. 17:26,365 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='yewu1122-restserver-leader'}. 17:26,373 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected a65df5ac-5139-4e81-b380-e1e4a268574d for yewu1122-restserver-leader. 17:26,389 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/rpc/resourcemanager_0 . 17:26,402 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector yewu1122-dispatcher-leader with lock identity 535eb66e-9240-40dd-90a7-a29ef88b8475. 17:26,409 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='yewu1122-dispatcher-leader'}. 17:26,410 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector yewu1122-resourcemanager-leader with lock identity 535eb66e-9240-40dd-90a7-a29ef88b8475. 17:26,465 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected a65df5ac-5139-4e81-b380-e1e4a268574d for yewu1122-dispatcher-leader. 17:26,468 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='yewu1122-resourcemanager-leader'}. 17:26,470 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='yewu1122-resourcemanager-leader'}. 17:26,475 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected a65df5ac-5139-4e81-b380-e1e4a268574d for yewu1122-resourcemanager-leader. 17:26,475 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with KubernetesLeaderRetrievalDriver{configMapName='yewu1122-dispatcher-leader'}. 17:31,848 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 535eb66e-9240-40dd-90a7-a29ef88b8475 for yewu1122-restserver-leader. 17:34,306 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 535eb66e-9240-40dd-90a7-a29ef88b8475 for yewu1122-dispatcher-leader. 17:34,318 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Start SessionDispatcherLeaderProcess. 17:34,319 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Recover all persisted job graphs. 17:34,325 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Retrieved job ids [9903fa9b42ff19c235749828ffcacd33, 9f0419e9786f419eb4eb39291788617d, bf16bea72af2424a6823c1723464c893] from KubernetesStateHandleStore{configMapName='yewu1122-dispatcher-leader'} 17:34,326 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Trying to recover job with job id 9903fa9b42ff19c235749828ffcacd33. 17:34,604 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Recovered JobGraph(jobId: 9903fa9b42ff19c235749828ffcacd33). 17:34,604 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Trying to recover job with job id 9f0419e9786f419eb4eb39291788617d. 17:34,733 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 535eb66e-9240-40dd-90a7-a29ef88b8475 for yewu1122-resourcemanager-leader. 17:34,741 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - ResourceManager akka.tcp://flink@172.19.89.67:6123/user/rpc/resourcemanager_0 was granted leadership with fencing token adee8f94ea359338580a61f9c2ca45a4 17:34,744 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] - Starting the SlotManager. 17:34,879 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Recovered JobGraph(jobId: 9f0419e9786f419eb4eb39291788617d). 17:34,879 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Trying to recover job with job id bf16bea72af2424a6823c1723464c893. 17:34,909 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 172.19.89.42:6122-6b1b56 (akka.tcp://flink@172.19.89.42:6122/user/rpc/taskmanager_0) at ResourceManager 17:34,913 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 172.19.90.99:6122-c9c6cd (akka.tcp://flink@172.19.90.99:6122/user/rpc/taskmanager_0) at ResourceManager 17:34,914 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 172.19.88.24:6122-4fae3a (akka.tcp://flink@172.19.88.24:6122/user/rpc/taskmanager_0) at ResourceManager 17:34,914 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering TaskManager with ResourceID 172.19.90.100:6122-107f28 (akka.tcp://flink@172.19.90.100:6122/user/rpc/taskmanager_0) at ResourceManager 17:34,921 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Recovered JobGraph(jobId: bf16bea72af2424a6823c1723464c893). 17:34,921 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Successfully recovered 3 persisted job graphs. 17:34,928 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_1 . 17:35,121 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector yewu1122-9f0419e9786f419eb4eb39291788617d-jobmanager-leader with lock identity 535eb66e-9240-40dd-90a7-a29ef88b8475. 17:35,128 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='yewu1122-9f0419e9786f419eb4eb39291788617d-jobmanager-leader'}. 17:35,137 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_2 . 17:35,142 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job State machine job (9f0419e9786f419eb4eb39291788617d). 17:35,143 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 535eb66e-9240-40dd-90a7-a29ef88b8475 for yewu1122-9f0419e9786f419eb4eb39291788617d-jobmanager-leader. 17:35,165 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647, backoffTimeMS=1000) for State machine job (9f0419e9786f419eb4eb39291788617d). 17:35,193 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job State machine job (9f0419e9786f419eb4eb39291788617d). 17:35,193 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 17:35,207 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 1 ms 17:35,221 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880) 17:35,232 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Recovering checkpoints from KubernetesStateHandleStore{configMapName='yewu1122-9f0419e9786f419eb4eb39291788617d-jobmanager-leader'}. 17:35,239 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found 1 checkpoints in KubernetesStateHandleStore{configMapName='yewu1122-9f0419e9786f419eb4eb39291788617d-jobmanager-leader'}. 17:35,239 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to fetch 1 checkpoints from storage. 17:35,239 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to retrieve checkpoint 21842. 17:35,455 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job 9f0419e9786f419eb4eb39291788617d from Checkpoint 21842 @ 1621999016577 for 9f0419e9786f419eb4eb39291788617d located at <checkpoint-not-externally-addressable>. 17:35,467 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master state to restore 17:35,469 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@4611a2be for State machine job (9f0419e9786f419eb4eb39291788617d). 17:35,484 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job State machine job (9f0419e9786f419eb4eb39291788617d) was granted leadership with session id 3174573b-51e3-4aba-a6a2-78990799c527 at akka.tcp://flink@172.19.89.67:6123/user/rpc/jobmanager_2. 17:36,046 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector yewu1122-9903fa9b42ff19c235749828ffcacd33-jobmanager-leader with lock identity 535eb66e-9240-40dd-90a7-a29ef88b8475. 17:36,070 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 535eb66e-9240-40dd-90a7-a29ef88b8475 for yewu1122-9903fa9b42ff19c235749828ffcacd33-jobmanager-leader. 17:36,107 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='yewu1122-9903fa9b42ff19c235749828ffcacd33-jobmanager-leader'}. 17:36,109 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_3 . 17:36,111 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job insert-into_iceberg_catalog.tracking_log.device_app_version (9903fa9b42ff19c235749828ffcacd33). 17:36,113 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=100, backoffTimeMS=20000) for insert-into_iceberg_catalog.tracking_log.device_app_version (9903fa9b42ff19c235749828ffcacd33). 17:36,116 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job insert-into_iceberg_catalog.tracking_log.device_app_version (9903fa9b42ff19c235749828ffcacd33). 17:36,116 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 17:36,117 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 0 ms 17:36,122 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using job/cluster config to configure application-defined state backend: File State Backend (checkpoints: 'hdfs:/flink/blblbl/cdc_device_app_version', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: -1) 17:36,123 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using application-defined state backend: File State Backend (checkpoints: 'hdfs:/flink/blblbl/cdc_device_app_version', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480) 17:36,204 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Recovering checkpoints from KubernetesStateHandleStore{configMapName='yewu1122-9903fa9b42ff19c235749828ffcacd33-jobmanager-leader'}. 17:36,210 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found 1 checkpoints in KubernetesStateHandleStore{configMapName='yewu1122-9903fa9b42ff19c235749828ffcacd33-jobmanager-leader'}. 17:36,210 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to fetch 1 checkpoints from storage. 17:36,210 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to retrieve checkpoint 17. 17:36,459 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job 9903fa9b42ff19c235749828ffcacd33 from Checkpoint 17 @ 1621998999693 for 9903fa9b42ff19c235749828ffcacd33 located at hdfs:/flink/blblbl/cdc_device_app_version/9903fa9b42ff19c235749828ffcacd33/chk-17. 17:36,461 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master state to restore 17:36,461 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@3579d6c2 for insert-into_iceberg_catalog.tracking_log.device_app_version (9903fa9b42ff19c235749828ffcacd33). 17:36,468 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job insert-into_iceberg_catalog.tracking_log.device_app_version (9903fa9b42ff19c235749828ffcacd33) was granted leadership with session id 956e3280-bfed-4d9b-bfe3-0c2982dd8e57 at akka.tcp://flink@172.19.89.67:6123/user/rpc/jobmanager_3. 17:36,768 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - Create KubernetesLeaderElector yewu1122-bf16bea72af2424a6823c1723464c893-jobmanager-leader with lock identity 535eb66e-9240-40dd-90a7-a29ef88b8475. 17:36,775 INFO org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] - Starting DefaultLeaderElectionService with KubernetesLeaderElectionDriver{configMapName='yewu1122-bf16bea72af2424a6823c1723464c893-jobmanager-leader'}. 17:36,778 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_4 . 17:36,778 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Initializing job insert-into_iceberg_catalog.tracking_log.risk_check_result (bf16bea72af2424a6823c1723464c893). 17:36,779 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=100, backoffTimeMS=20000) for insert-into_iceberg_catalog.tracking_log.risk_check_result (bf16bea72af2424a6823c1723464c893). 17:36,780 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running initialization on master for job insert-into_iceberg_catalog.tracking_log.risk_check_result (bf16bea72af2424a6823c1723464c893). 17:36,780 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Successfully ran initialization on master in 0 ms. 17:36,781 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 pipelined regions in 0 ms 17:36,782 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using job/cluster config to configure application-defined state backend: File State Backend (checkpoints: 'hdfs:/flink/blblbl/cdc_risk_check_result', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: -1) 17:36,782 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using application-defined state backend: File State Backend (checkpoints: 'hdfs:/flink/blblbl/cdc_risk_check_result', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480) 17:36,846 INFO org.apache.flink.kubernetes.kubeclient.resources.KubernetesLeaderElector [] - New leader elected 535eb66e-9240-40dd-90a7-a29ef88b8475 for yewu1122-bf16bea72af2424a6823c1723464c893-jobmanager-leader. 17:36,864 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Recovering checkpoints from KubernetesStateHandleStore{configMapName='yewu1122-bf16bea72af2424a6823c1723464c893-jobmanager-leader'}. 17:36,921 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Found 1 checkpoints in KubernetesStateHandleStore{configMapName='yewu1122-bf16bea72af2424a6823c1723464c893-jobmanager-leader'}. 17:36,922 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to fetch 1 checkpoints from storage. 17:36,922 INFO org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore [] - Trying to retrieve checkpoint 16. 17:37,130 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job bf16bea72af2424a6823c1723464c893 from Checkpoint 16 @ 1621998993016 for bf16bea72af2424a6823c1723464c893 located at hdfs:/flink/blblbl/cdc_risk_check_result/bf16bea72af2424a6823c1723464c893/chk-16. 17:37,131 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master state to restore 17:37,131 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@77ed6b22 for insert-into_iceberg_catalog.tracking_log.risk_check_result (bf16bea72af2424a6823c1723464c893). 17:37,137 INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl [] - JobManager runner for job insert-into_iceberg_catalog.tracking_log.risk_check_result (bf16bea72af2424a6823c1723464c893) was granted leadership with session id 4d39cb39-26c6-4259-8b50-23ed84bac6f4 at akka.tcp://flink@172.19.89.67:6123/user/rpc/jobmanager_4. 17:51,902 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler [] - Unhandled exception. org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(934936333dcfc172032cab785b424a3b, LocalRpcInvocation(requestClusterOverview(Time))) because the fencing token 934936333dcfc172032cab785b424a3b did not match the expected fencing token acbe7f0ced9f66de89ff2bdad0704770. at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:88) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[flink-dist_2.11-1.12.2.jar:1.12.2] 17:53,973 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler [] - Unhandled exception. org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(934936333dcfc172032cab785b424a3b, LocalRpcInvocation(requestClusterOverview(Time))) because the fencing token 934936333dcfc172032cab785b424a3b did not match the expected fencing token acbe7f0ced9f66de89ff2bdad0704770. at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:88) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.11-1.12.2.jar:1.12.2] 17:53,975 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled exception. org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(934936333dcfc172032cab785b424a3b, LocalRpcInvocation(requestMultipleJobDetails(Time))) because the fencing token 934936333dcfc172032cab785b424a3b did not match the expected fencing token acbe7f0ced9f66de89ff2bdad0704770. at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:88) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.11-1.12.2.jar:1.12.2] 17:56,975 ERROR org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler [] - Unhandled exception. org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(934936333dcfc172032cab785b424a3b, LocalRpcInvocation(requestClusterOverview(Time))) because the fencing token 934936333dcfc172032cab785b424a3b did not match the expected fencing token acbe7f0ced9f66de89ff2bdad0704770. at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:88) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.11-1.12.2.jar:1.12.2] 17:56,977 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled exception. org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token mismatch: Ignoring message LocalFencedMessage(934936333dcfc172032cab785b424a3b, LocalRpcInvocation(requestMultipleJobDetails(Time))) because the fencing token 934936333dcfc172032cab785b424a3b did not match the expected fencing token acbe7f0ced9f66de89ff2bdad0704770. at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:88) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.Actor$class.aroundReceive(Actor.scala:517) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.11-1.12.2.jar:1.12.2] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.11-1.12.2.jar:1.12.2] 18:02,024 INFO org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor [] - The rpc endpoint org.apache.flink.runtime.jobmaster.JobMaster has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.RemoteFencedMessage until processing is started. 18:02,434 INFO org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor [] - The rpc endpoint org.apache.flink.runtime.jobmaster.JobMaster has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.RemoteFencedMessage until processing is started. 18:02,572 INFO org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor [] - The rpc endpoint org.apache.flink.runtime.jobmaster.JobMaster has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.RemoteFencedMessage until processing is started.
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |