TM get killed/disconnected after a while

[Hao Sun]

Hi, I am running Flink 1.3.2 on kubernetes, I am not sure why sometime one of my TM is killed, is there a way to debug this? Thanks

===== Logs ====

2017-10-05 22:36:42,631 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at fps-flink-taskmanager-2384273947-9n4kc (akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274/user/taskmanager) as 330ff7eeaabfe2b7289fee4a0e36c4b2. Current number of registered hosts is 2. Current number of alive task slots is 2.

2017-10-05 22:37:04,974 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: KafkaSource(maxwell.users) -> MaxwellFilter->Maxwell(maxwell.users) -> FixedDelayWatermark(maxwell.users) -> MaxwellFPSEvent->InfluxDBData(maxwell.users) -> (Sink: influxdbSink(maxwell.users), Sink: PrintSink(maxwell.users)) (1/1) (attempt #0) to fps-flink-taskmanager-2384273947-9n4kc

2017-10-06 06:08:55,657 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Disassociated]

2017-10-06 06:08:55,832 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:09:01,232 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:09:03,416 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:11,174 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:11,440 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:09:21,232 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:09:27,460 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:09:31,173 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:41,179 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:09:51,174 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:57,475 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:10:01,179 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:10:06,173 WARN  akka.remote.RemoteWatcher                                     - Detected unreachable: [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]

2017-10-06 06:10:06,177 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Task manager akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274/user/taskmanager terminated.

java.lang.Exception: TaskManager was lost/killed: 55d3143ccecec7878f7df169208795d0 @ fps-flink-taskmanager-2384273947-9n4kc (dataPort=37448)

java.lang.Exception: TaskManager was lost/killed: 55d3143ccecec7878f7df169208795d0 @ fps-flink-taskmanager-2384273947-9n4kc (dataPort=37448)

2017-10-06 06:10:06,188 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:10:06,240 INFO  org.apache.flink.runtime.instance.InstanceManager             - Unregistered task manager fps-flink-taskmanager-2384273947-9n4kc/10.225.132.78. Number of registered task managers 3. Number of available slots 3.

2017-10-06 06:10:16,247 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:10:26,284 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:10:27,495 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

[Patrick Lucas]

Hi,

Can you provide a bit more info about your setup, such as what Kubernetes resources you are using? (Deployments, Service)

Is the pod running the taskmanager killed by Kubernetes or does it fail? Can you provide the output of kubectl describe pod <pod> and kubectl logs <pod> of the taskmanager pod that exited?

On Fri, Oct 6, 2017 at 8:16 PM, Hao Sun <[hidden email]> wrote:

Hi, I am running Flink 1.3.2 on kubernetes, I am not sure why sometime one of my TM is killed, is there a way to debug this? Thanks

===== Logs ====

2017-10-05 22:36:42,631 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at fps-flink-taskmanager-2384273947-9n4kc (akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274/user/taskmanager) as 330ff7eeaabfe2b7289fee4a0e36c4b2. Current number of registered hosts is 2. Current number of alive task slots is 2.

2017-10-05 22:37:04,974 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: KafkaSource(maxwell.users) -> MaxwellFilter->Maxwell(maxwell.users) -> FixedDelayWatermark(maxwell.users) -> MaxwellFPSEvent->InfluxDBData(maxwell.users) -> (Sink: influxdbSink(maxwell.users), Sink: PrintSink(maxwell.users)) (1/1) (attempt #0) to fps-flink-taskmanager-2384273947-9n4kc

2017-10-06 06:08:55,657 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Disassociated]

2017-10-06 06:08:55,832 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:09:01,232 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:09:03,416 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:11,174 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:11,440 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:09:21,232 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:09:27,460 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:09:31,173 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:41,179 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:09:51,174 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:09:57,475 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]

2017-10-06 06:10:01,179 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:10:06,173 WARN  akka.remote.RemoteWatcher                                     - Detected unreachable: [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]

2017-10-06 06:10:06,177 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Task manager akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274/user/taskmanager terminated.

java.lang.Exception: TaskManager was lost/killed: 55d3143ccecec7878f7df169208795d0 @ fps-flink-taskmanager-2384273947-9n4kc (dataPort=37448)

java.lang.Exception: TaskManager was lost/killed: 55d3143ccecec7878f7df169208795d0 @ fps-flink-taskmanager-2384273947-9n4kc (dataPort=37448)

2017-10-06 06:10:06,188 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc]

2017-10-06 06:10:06,240 INFO  org.apache.flink.runtime.instance.InstanceManager             - Unregistered task manager fps-flink-taskmanager-2384273947-9n4kc/10.225.132.78. Number of registered task managers 3. Number of available slots 3.

2017-10-06 06:10:16,247 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:10:26,284 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]] Caused by: [fps-flink-taskmanager-2384273947-9n4kc: Name does not resolve]

2017-10-06 06:10:27,495 WARN  Remoting                                                      - Tried to associate with unreachable remote address [akka.tcp://flink@fps-flink-taskmanager-2384273947-9n4kc:40274]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]