(DEPRECATED) Apache Flink User Mailing List archive.

Connecting to remote task manager failed

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Charith Wickramarachchi
Reply | Threaded
Open this post in threaded view
♦
♦
|

Connecting to remote task manager failed

Charith Wickramarachchi
Hi All, 

I m getting an exception when running a Gelly task using Pregel model. It complains that the remote task manager might be lost. But task managers seem to be active based on the flink dashboard.  Also, other tasks run fine without an issue. 


Following is the summary of exception trace.  I have attached the full trace as well. It will be great if you can provide any directions to identify the issue. 


Flink version: flink-1.1.3
Java: 1.7

Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
Caused by: java.io.IOException: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.waitForChannel(PartitionRequestClientFactory.java:196)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.access$000(PartitionRequestClientFactory.java:131)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:83)
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:60)
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:118)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:394)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:413)
at org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:87)
at org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:42)
at org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:59)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:973)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:796)
Caused by: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:219)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:131)

Thanks,
Charith


--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions

error.txt (10K) Download Attachment
Charith Wickramarachchi
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Connecting to remote task manager failed

Charith Wickramarachchi
I did some more digging. It seems the CoGroup operation failed in one of the workers. But I do not face this issue when running other tasks. 

Thanks,
Charith

On Tue, Jul 25, 2017 at 2:06 PM, Charith Wickramarachchi <[hidden email]> wrote:
Hi All, 

I m getting an exception when running a Gelly task using Pregel model. It complains that the remote task manager might be lost. But task managers seem to be active based on the flink dashboard.  Also, other tasks run fine without an issue. 


Following is the summary of exception trace.  I have attached the full trace as well. It will be great if you can provide any directions to identify the issue. 


Flink version: flink-1.1.3
Java: 1.7

Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
Caused by: java.io.IOException: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.waitForChannel(PartitionRequestClientFactory.java:196)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.access$000(PartitionRequestClientFactory.java:131)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:83)
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:60)
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:118)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:394)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:413)
at org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:87)
at org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:42)
at org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:59)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:973)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:796)
Caused by: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:219)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:131)

Thanks,
Charith


--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions



--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Connecting to remote task manager failed

Fabian Hueske-2
Hi,

Do you have an exception for the the CoGroup failure?

Best, Fabian

2017-07-26 3:32 GMT+02:00 Charith Wickramarachchi <[hidden email]>:
I did some more digging. It seems the CoGroup operation failed in one of the workers. But I do not face this issue when running other tasks. 

Thanks,
Charith

On Tue, Jul 25, 2017 at 2:06 PM, Charith Wickramarachchi <[hidden email]> wrote:
Hi All, 

I m getting an exception when running a Gelly task using Pregel model. It complains that the remote task manager might be lost. But task managers seem to be active based on the flink dashboard.  Also, other tasks run fine without an issue. 


Following is the summary of exception trace.  I have attached the full trace as well. It will be great if you can provide any directions to identify the issue. 


Flink version: flink-1.1.3
Java: 1.7

Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
Caused by: java.io.IOException: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.waitForChannel(PartitionRequestClientFactory.java:196)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.access$000(PartitionRequestClientFactory.java:131)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:83)
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:60)
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:118)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:394)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:413)
at org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:87)
at org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:42)
at org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:59)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:973)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:796)
Caused by: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:219)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:131)

Thanks,
Charith


--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions



--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions
Charith Wickramarachchi
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Connecting to remote task manager failed

Charith Wickramarachchi
Hi Fabian, 

I see the following exceptions in worker logs (Exception trace is similar to the one I attached above). I m wondering if its a network configuration issue because it refers to a 127.0.1.1 address.  

Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:43061' has failed. This might indicate that the remote task manager has been lost.
        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:619)
        at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1098)
        at org.apache.flink.runtime.operators.BatchTask.initLocalStrategies(BatchTask.java:831)
        at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:327)
        ... 2 more


Thanks,
Charith

On Wed, Jul 26, 2017 at 4:29 AM, Fabian Hueske <[hidden email]> wrote:
Hi,

Do you have an exception for the the CoGroup failure?

Best, Fabian

2017-07-26 3:32 GMT+02:00 Charith Wickramarachchi <[hidden email]>:
I did some more digging. It seems the CoGroup operation failed in one of the workers. But I do not face this issue when running other tasks. 

Thanks,
Charith

On Tue, Jul 25, 2017 at 2:06 PM, Charith Wickramarachchi <[hidden email]> wrote:
Hi All, 

I m getting an exception when running a Gelly task using Pregel model. It complains that the remote task manager might be lost. But task managers seem to be active based on the flink dashboard.  Also, other tasks run fine without an issue. 


Following is the summary of exception trace.  I have attached the full trace as well. It will be great if you can provide any directions to identify the issue. 


Flink version: flink-1.1.3
Java: 1.7

Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
Caused by: java.io.IOException: Connecting the channel failed: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.waitForChannel(PartitionRequestClientFactory.java:196)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.access$000(PartitionRequestClientFactory.java:131)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:83)
at org.apache.flink.runtime.io.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:60)
at org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:118)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:394)
at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:413)
at org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:87)
at org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:42)
at org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:59)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ReadingThread.go(UnilateralSortMerger.java:973)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:796)
Caused by: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connecting to remote task manager + 'worker/127.0.1.1:44310' has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:219)
at org.apache.flink.runtime.io.network.netty.PartitionRequestClientFactory$ConnectingChannel.operationComplete(PartitionRequestClientFactory.java:131)

Thanks,
Charith


--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions



--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions




--
Charith Dhanushka Wickramaarachchi

This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions
Free forum by Nabble Edit this page