Taskmanager fails to connect to Jobmanager [Could not find any IPv4 address that is not loopback or link-local. Using localhost address.]
Locked
Taskmanager fails to connect to Jobmanager [Could not find any IPv4 address that is not loopback or link-local. Using localhost address.]
 
|
Hi all, We were previously using RHEL for our Flink machines. I'm currently working on moving them over to Ubuntu. When I start the task manager, it fails to connect to the job manager with the following
message - 2020-01-16 10:54:42,777 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading
JobManager. 2020-01-16 10:54:42,778 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics 2020-01-16 10:54:52,780 WARN org.apache.flink.runtime.net.ConnectionUtils - Could not find any IPv4 address that is not loopback or link-local. Using localhost
address. The network interface on the machine looks like this -
ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001 inet 10.16.75.30 netmask 255.255.255.128 broadcast 10.16.75.127 ether 02:f1:8b:34:75:51 txqueuelen 1000 (Ethernet) RX packets 69370 bytes 80369110 (80.3 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 28787 bytes 2898540 (2.8 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 loop txqueuelen 1000 (Local Loopback) RX packets 9562 bytes 1596138 (1.5 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 9562 bytes 1596138 (1.5 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Note: On RHEL, the primary network interface was eth0. Could this be the issue? Here's the full task manager log -
https://paste.ubuntu.com/p/vgh96FHzRq/ Thanks Harshith |
|
|
Hi Kumar Bolar, Harshith, Could you please check the jobmanager log to find out what address the akka is listening? Also the address could be used to connected to the jobmanager on the taskmanger machine. BTW, if you could share the debug level logs of jobmanger and taskmanger. It will help a lot to find the root cause. Best, Yang Kumar Bolar, Harshith <[hidden email]> 于2020年1月16日周四 下午7:10写道：
|
Re: Taskmanager fails to connect to Jobmanager [Could not find any IPv4 address that is not loopback or link-local. Using localhost address.]
|
|
Hi, Harshith
As a supplementary note to Yang, the issue seems to be that something went wrong when trying to connect to the ResourceManager. There may be two possibilities, the leader of ResourceManager does not write the znode or the TaskExecutor fails to connect to it. If you turn on the DEBUG log, it will help a lot. Also, you could watch the content znode "/leader/resource_manager_lock" of ZooKeeper. Best, Yangze Guo On Fri, Jan 17, 2020 at 5:11 PM Yang Wang <[hidden email]> wrote: > > Hi Kumar Bolar, Harshith, > > Could you please check the jobmanager log to find out what address the akka is listening? > Also the address could be used to connected to the jobmanager on the taskmanger machine. > > BTW, if you could share the debug level logs of jobmanger and taskmanger. It will help a lot to find > the root cause. > > > Best, > Yang > > Kumar Bolar, Harshith <[hidden email]> 于2020年1月16日周四 下午7:10写道： >> >> Hi all, >> >> >> >> We were previously using RHEL for our Flink machines. I'm currently working on moving them over to Ubuntu. When I start the task manager, it fails to connect to the job manager with the following message - >> >> >> >> 2020-01-16 10:54:42,777 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager. >> >> 2020-01-16 10:54:42,778 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics >> >> 2020-01-16 10:54:52,780 WARN org.apache.flink.runtime.net.ConnectionUtils - Could not find any IPv4 address that is not loopback or link-local. Using localhost address. >> >> >> >> The network interface on the machine looks like this - >> >> >> >> >> >> ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001 >> >> inet 10.16.75.30 netmask 255.255.255.128 broadcast 10.16.75.127 >> >> ether 02:f1:8b:34:75:51 txqueuelen 1000 (Ethernet) >> >> RX packets 69370 bytes 80369110 (80.3 MB) >> >> RX errors 0 dropped 0 overruns 0 frame 0 >> >> TX packets 28787 bytes 2898540 (2.8 MB) >> >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> >> >> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >> >> inet 127.0.0.1 netmask 255.0.0.0 >> >> loop txqueuelen 1000 (Local Loopback) >> >> RX packets 9562 bytes 1596138 (1.5 MB) >> >> RX errors 0 dropped 0 overruns 0 frame 0 >> >> TX packets 9562 bytes 1596138 (1.5 MB) >> >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> >> >> >> >> Note: On RHEL, the primary network interface was eth0. Could this be the issue? >> >> >> >> Here's the full task manager log - https://paste.ubuntu.com/p/vgh96FHzRq/ >> >> >> >> Thanks >> >> Harshith |
Re: Re: Taskmanager fails to connect to Jobmanager [Could not find any IPv4 address that is not loopback or link-local. Using localhost address.]
|
|
Thank you, Yangze and Yang.
Turns out the high-availability.cluster-id parameter on the TM and JM were different. After updating it, the issue went away. On 17/01/20, 3:14 PM, "Yangze Guo" <[hidden email]> wrote: Hi, Harshith As a supplementary note to Yang, the issue seems to be that something went wrong when trying to connect to the ResourceManager. There may be two possibilities, the leader of ResourceManager does not write the znode or the TaskExecutor fails to connect to it. If you turn on the DEBUG log, it will help a lot. Also, you could watch the content znode "/leader/resource_manager_lock" of ZooKeeper. Best, Yangze Guo On Fri, Jan 17, 2020 at 5:11 PM Yang Wang <[hidden email]> wrote: > > Hi Kumar Bolar, Harshith, > > Could you please check the jobmanager log to find out what address the akka is listening? > Also the address could be used to connected to the jobmanager on the taskmanger machine. > > BTW, if you could share the debug level logs of jobmanger and taskmanger. It will help a lot to find > the root cause. > > > Best, > Yang > > Kumar Bolar, Harshith <[hidden email]> 于2020年1月16日周四 下午7:10写道： >> >> Hi all, >> >> >> >> We were previously using RHEL for our Flink machines. I'm currently working on moving them over to Ubuntu. When I start the task manager, it fails to connect to the job manager with the following message - >> >> >> >> 2020-01-16 10:54:42,777 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager. >> >> 2020-01-16 10:54:42,778 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics >> >> 2020-01-16 10:54:52,780 WARN org.apache.flink.runtime.net.ConnectionUtils - Could not find any IPv4 address that is not loopback or link-local. Using localhost address. >> >> >> >> The network interface on the machine looks like this - >> >> >> >> >> >> ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001 >> >> inet https://urldefense.proofpoint.com/v2/url?u=http-3A__10.16.75.30&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=53OA1njQf1iaaB9btpV8bgi7qYya9rwWK9DUUr9A580&e= netmask https://urldefense.proofpoint.com/v2/url?u=http-3A__255.255.255.128&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=VAkjHsvEbPDirc3_I_ZhOOrimAfIYYsGUUEVaC1rhH8&e= broadcast https://urldefense.proofpoint.com/v2/url?u=http-3A__10.16.75.127&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=hXAq7oqzjzyE4mGfBDaQQhn-LhoxC7tjLRTzYvNufxE&e= >> >> ether 02:f1:8b:34:75:51 txqueuelen 1000 (Ethernet) >> >> RX packets 69370 bytes 80369110 (80.3 MB) >> >> RX errors 0 dropped 0 overruns 0 frame 0 >> >> TX packets 28787 bytes 2898540 (2.8 MB) >> >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> >> >> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >> >> inet https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=qzQ1Wkjhm6A1Y5WVp7oWupNN6xESCglqAofLiJnyXXg&e= netmask https://urldefense.proofpoint.com/v2/url?u=http-3A__255.0.0.0&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=bdn272PCxu0YYSd6dH2BPxITorcdno5flP1nOB379ns&e= >> >> loop txqueuelen 1000 (Local Loopback) >> >> RX packets 9562 bytes 1596138 (1.5 MB) >> >> RX errors 0 dropped 0 overruns 0 frame 0 >> >> TX packets 9562 bytes 1596138 (1.5 MB) >> >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> >> >> >> >> Note: On RHEL, the primary network interface was eth0. Could this be the issue? >> >> >> >> Here's the full task manager log - https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.ubuntu.com_p_vgh96FHzRq_&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=xslfIsmeNZvi6en0rEcsuZ-0ODtAxVYaNgAJViglZDY&e= >> >> >> >> Thanks >> >> Harshith |
| Free forum by Nabble | Edit this page |