(DEPRECATED) Apache Flink User Mailing List archive.

Fwd: HA : My job didn't restart even if task manager restarted.

Classic

List

Threaded

6 messages Options

sunny yun

Fwd: HA : My job didn't restart even if task manager restarted.

I have 4 nodes for flink cluster using flink 1.3.2 and hadoop 2.6.5. All configurations are same in 4 nodes.

Running yarn session and flink job, I killed node#2(flink-02)’s TM(YarnTaskManager) process.

After resoucemanager make new TM container, job still failed status.

But when I killed JM(YarnApplicationMasterRunner) process, all behavior seems ok. My job restarted well.

I’m using DFS for storageDir.

Flink configuration file and log files are attached. If any other information needed, please let me know.

Here is use case, -------------------------------------------------------------------------------------------------------

1. start yarn session

flink-01 ~]$ flink/bin/yarn-session.sh -n 4 -s 8 -nm flink -d

2. submit job

flink-01 ~]$ flink/bin/flink run -d -p 3 -c a.b.c.Application abc.jar

3. find TM pid and kill it

flink-02 ~]$ jps | grep YarnTaskManager

4022 YarnTaskManager

flink-02 ~]$ kill 4022

And JM logs -------------------------------------------------------------------------------------------------------

2017-09-04 00:50:57,830 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@flink-02:45104] has failed, address is now gated for [5000] ms. Reason: [Disassociated]

2017-09-04 00:50:58,013 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Container container_1504449399430_0005_01_000004 failed. Exit status: 143

2017-09-04 00:50:58,014 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Diagnostics for container container_1504449399430_0005_01_000004 in state COMPLETE : exitStatus=143 diagnostics=Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

Killed by external signal

2017-09-04 00:50:58,014 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Total number of failed containers so far: 1

2017-09-04 00:50:58,014 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Received new container: container_1504449399430_0005_01_000005 - Remaining pending container requests: 0

2017-09-04 00:50:58,014 INFO org.apache.flink.yarn.YarnFlinkResourceManager - Launching TaskManager in container ContainerInLaunch @ 1504453858014: Container: [ContainerId: container_1504449399430_0005_01_000005, NodeId: flink-02:34821, NodeHttpAddress: flink-02:8042, Resource: <memory:8192, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.1.0.5:34821 }, ] on host flink-02

2017-09-04 00:50:58,015 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : flink-02:34821

2017-09-04 00:50:58,015 INFO org.apache.flink.yarn.YarnJobManager - Task manager akka.tcp://flink@flink-02:45104/user/taskmanager terminated.

2017-09-04 00:50:58,016 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: src -> Service: vps -> Sink: snk (3/3) (33b33a9c15eb0107b320e375374ac07f) switched from RUNNING to FAILED.

java.lang.Exception: TaskManager was lost/killed: container_1504449399430_0005_01_000004 @ flink-02 (dataPort=42381)

at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:217)

at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533)

at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:192)

at org.apache.flink.runtime.instance.Instance.markDead(Instance.java:167)

at org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:212)

at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$handleTaskManagerTerminated(JobManager.scala:1228)

at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:474)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.clusterframework.ContaineredJobManager$$anonfun$handleContainerMessage$1.applyOrElse(ContaineredJobManager.scala:100)

at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)

at org.apache.flink.yarn.YarnJobManager$$anonfun$handleYarnShutdown$1.applyOrElse(YarnJobManager.scala:103)

at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)

at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)

at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)

at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)

at akka.actor.Actor$class.aroundReceive(Actor.scala:467)

at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:125)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

at akka.actor.ActorCell.invoke(ActorCell.scala:487)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)

at akka.dispatch.Mailbox.run(Mailbox.scala:220)

at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

2017-09-04 00:50:58,018 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job vp (6e73f6babf36c9321efc0524a82dede1) switched from state RUNNING to FAILING.