(DEPRECATED) Apache Flink User Mailing List archive.

JM/TM Disassociated

Classic

List

Threaded

1 message

miki haiat

JM/TM Disassociated

Hi,

I have some simple elesticsearch sink on mesos with 6 TM ,

but for some reason after few hours JM/TM is Disassociated and killed all other TM as well .

this is were everything collapse

2018-04-29 08:21:00,457 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
2018-04-29 08:21:06,501 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123

2018-04-29 08:21:00,457 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
2018-04-29 08:21:06,501 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:06,503 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:16,501 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:16,504 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:26,500 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:26,501 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:31,124 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager: Old JobManager lost its leadership.
2018-04-29 08:21:31,124 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Cancelling all computations and discarding all cached data.
2018-04-29 08:21:31,125 INFO  org.apache.flink.runtime.taskmanager.Task                     - Attempting to fail task externally Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae).
2018-04-29 08:21:31,127 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae) switched from RUNNING to FAILED.
java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager: Old JobManager lost its leadership.
	at org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:1073)
	at org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleJobManagerLeaderAddress(TaskManager.scala:1467)
	at org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:277)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:49)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
	at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
	at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
	at org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:121)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-04-29 08:21:31,129 INFO  org.apache.flink.runtime.taskmanager.Task                     - Triggering cancellation of task code Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae).
2018-04-29 08:21:31,130 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Disassociating from JobManager
2018-04-29 08:21:31,141 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
2018-04-29 08:21:31,145 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #17
2018-04-29 08:21:31,145 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae).
2018-04-29 08:21:31,173 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #1
2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #2
2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #3
2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #4
2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #5
2018-04-29 08:21:31,172 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #6
2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #7
2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #8
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #9
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #10
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #11
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #12
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #13
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #14
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #15
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #16
2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector  - Interrupted while wait for resources to be released #17
2018-04-29 08:21:31,211 INFO  org.apache.flink.runtime.taskmanager.Task                     - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae) [FAILED]
2018-04-29 08:21:31,520 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:31,520 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:36,540 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:36,541 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:41,559 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:41,560 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:46,581 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:46,581 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:51,602 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:51,602 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:21:56,621 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:21:56,621 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:01,640 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:01,649 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:06,659 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:06,660 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:11,680 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:11,680 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:16,700 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:16,701 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:21,719 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:21,720 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:26,740 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:26,740 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:31,760 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:31,760 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:36,780 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:36,781 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:41,800 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:41,800 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:46,820 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:46,820 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:51,840 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:51,841 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:22:56,860 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:22:56,860 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:01,879 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:01,880 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:06,901 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:06,901 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:11,919 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:11,919 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:16,939 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:16,940 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:21,959 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:21,960 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:26,984 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:26,984 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:32,000 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:32,000 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:37,024 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:37,025 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:42,039 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:42,040 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:47,060 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:47,060 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:52,080 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:52,080 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:23:57,101 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:23:57,101 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:24:02,119 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:24:02,120 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:24:07,140 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:24:07,140 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:24:12,159 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:24:12,160 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:24:17,180 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:24:17,180 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:24:22,201 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:24:22,202 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:24:27,219 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 08:24:27,219 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 08:24:31,285 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Trying to register at JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)
2018-04-29 08:24:31,797 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Trying to register at JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager (attempt 2, timeout: 1000 milliseconds)
2018-04-29 08:24:32,240 ERROR akka.remote.Remoting                                          - Association to [akka.tcp://flink@10.1.70.116:6123] with UID [1855709371] irrecoverably failed. Quarantining address.
java.util.concurrent.TimeoutException: Delivery of system messages timed out and they were dropped.
	at akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoint.scala:346)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
	at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-04-29 08:24:32,817 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Trying to register at JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager (attempt 3, timeout: 2000 milliseconds)
2018-04-29 08:24:32,987 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Successful registration at JobManager (akka.tcp://flink@10.1.70.116:6123/user/jobmanager), starting network stack and library cache.
2018-04-29 08:24:32,987 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Determined BLOB server address to be /10.1.70.116:37931. Starting BLOB cache.
2018-04-29 08:24:32,987 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Created BLOB cache storage directory /tmp/blobStore-14e1221d-bb62-4e38-8c5a-4b19d003498b
2018-04-29 08:24:33,010 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Created BLOB cache storage directory /tmp/blobStore-69deafd1-b90e-4c44-acc5-282bf9690f00
2018-04-29 11:32:20,287 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
2018-04-29 11:32:28,030 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:32:28,031 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:32:38,030 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:32:38,031 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:32:48,031 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:32:48,031 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:32:51,117 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager: Old JobManager lost its leadership.
2018-04-29 11:32:51,117 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager  - Disassociating from JobManager
2018-04-29 11:32:51,118 INFO  org.apache.flink.runtime.blob.PermanentBlobCache              - Shutting down BLOB cache
2018-04-29 11:32:51,118 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
2018-04-29 11:32:53,050 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:32:53,051 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:32:58,069 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:32:58,069 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:03,090 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:03,091 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:08,109 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:08,110 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:13,132 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:13,132 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:18,149 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:18,150 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:23,170 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:23,170 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:28,189 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:28,190 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:33,210 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:33,210 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:38,230 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:38,230 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:43,249 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:43,250 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:48,269 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:48,272 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:53,290 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:53,291 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:33:58,311 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:33:58,312 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:03,329 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:03,330 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:08,352 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:08,352 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:13,373 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:13,377 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:18,399 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:18,400 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:23,420 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:23,421 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:28,440 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:28,440 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:33,459 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:33,459 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:38,481 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:38,481 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:43,499 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:43,500 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:48,519 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:48,519 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:53,539 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:53,540 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:34:58,566 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:34:58,567 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:03,591 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:03,591 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:08,610 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:08,610 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:13,629 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:13,630 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:18,650 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:18,651 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:23,672 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:23,673 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:28,689 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:28,690 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:33,709 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:33,716 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:38,729 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:38,731 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:43,750 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:43,751 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:48,769 WARN  akka.remote.transport.netty.NettyTransport                    - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
2018-04-29 11:35:48,770 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
2018-04-29 11:35:53,787 ERROR akka.remote.Remoting                                          - Association to [akka.tcp://flink@10.1.70.116:6123] with UID [-853636033] irrecoverably failed. Quarantining address.
java.util.concurrent.TimeoutException: Delivery of system messages timed out and they were dropped.
	at akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoint.scala:346)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
	at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-04-29 11:42:20,322 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosTaskManagerRunner  - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2018-04-29 11:42:20,337 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /tmp/flink-io-bf946b07-190d-4e16-a5dc-fa76bc8498db