Standalone cluster instability

Posted by Alexander Smirnov on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Standalone-cluster-instability-tp19091.html

Hello,

I've assembled a standalone cluster of 3 task managers and 3 job managers(and 3 ZK) following the instructions at 

https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/cluster_setup.html and https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/jobmanager_high_availability.html

It works ok, but randomly, task managers becomes unavailable. JobManager has exception like below in logs:

[2018-03-19 00:33:10,211] WARN Association with remote system [akka.tcp://flink@...:42413] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@...:42413]] Caused by: [Connection refused: qafdsflinkw811.nn.five9lab.com/10.5.61.124:42413] (akka.remote.ReliableDeliverySupervisor)

[2018-03-21 00:30:35,975] ERROR Association to [akka.tcp://flink@...:35320] with UID [1500204560] irrecoverably failed. Quarantining address. (akka.remote.Remoting)

java.util.concurrent.TimeoutException: Remote system has been silent for too long. (more than 48.0 hours)

        at akka.remote.ReliableDeliverySupervisor$$anonfun$idle$1.applyOrElse(Endpoint.scala:375)

        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)

        at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203)

        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)

        at akka.actor.ActorCell.invoke(ActorCell.scala:495)

        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)

        at akka.dispatch.Mailbox.run(Mailbox.scala:224)

        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)

        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I can't find a reason for this exception, any ideas?

Thank you,

Alex