"Failed to retrieve JobManager address" in Flink 1.1.1 with JM HA

[Hironori Ogibayashi]

Hello,

After I upgraded to 1.1.1, I am getting error when submitting job with
"flink run"

The command and result is like this. It has been working with Flink 1.0.3.

---
 % FLINK_CONF_DIR=~/opt/flink/conf ~/opt/flink/flink-1.1.1/bin/flink
run -c MyJob target/my-flink-job.jar

------------------------------------------------------------
 The program finished with the following exception:

java.lang.RuntimeException: Failed to retrieve JobManager address
        at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:244)
        at org.apache.flink.client.program.StandaloneClusterClient.getClusterIdentifier(StandaloneClusterClient.java:78)
        at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:887)
        at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)
        at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
        at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
Caused by: java.lang.IllegalArgumentException: hostname can't be null
        at java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
        at java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
        at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:242)
        ... 5 more
---

I am using JobManager HA and I set "recovery.mode: zookeeper",
recovery.zookeeper.quorum, recovery.zookeeper.path.root is my
flink-conf.yaml.
So, the client should be able to get JobManager address from zookeeper.
If I explicitly specify JobManager address with -m option, it works.

Am I missing something?

Regards,
Hironori Ogibayashi

[Ufuk Celebi]

You are right that this config key is not needed in this case.

The ClusterClient has been refactored between Flink 1.0 and 1.1 and
the config parsing might be too strict in this case. It expects the
IPC address to be set, which is not necessary as you say. It should be
very easy to fix for 1.1.2. Let's confirm that it is actually a bug
with Max and file an issue afterwards.

@Max: can you confirm whether this is correct?


On Tue, Aug 23, 2016 at 7:24 AM, Hironori Ogibayashi
<[hidden email]> wrote:

> Hello,
>
> After I upgraded to 1.1.1, I am getting error when submitting job with
> "flink run"
>
> The command and result is like this. It has been working with Flink 1.0.3.
>
> ---
>  % FLINK_CONF_DIR=~/opt/flink/conf ~/opt/flink/flink-1.1.1/bin/flink
> run -c MyJob target/my-flink-job.jar
>
> ------------------------------------------------------------
>  The program finished with the following exception:
>
> java.lang.RuntimeException: Failed to retrieve JobManager address
>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:244)
>         at org.apache.flink.client.program.StandaloneClusterClient.getClusterIdentifier(StandaloneClusterClient.java:78)
>         at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:887)
>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)
>         at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
> Caused by: java.lang.IllegalArgumentException: hostname can't be null
>         at java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
>         at java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:242)
>         ... 5 more
> ---
>
> I am using JobManager HA and I set "recovery.mode: zookeeper",
> recovery.zookeeper.quorum, recovery.zookeeper.path.root is my
> flink-conf.yaml.
> So, the client should be able to get JobManager address from zookeeper.
> If I explicitly specify JobManager address with -m option, it works.
>
> Am I missing something?
>
> Regards,
> Hironori Ogibayashi

[Maximilian Michels]

Hi!

Yes, this is a bug. However, there seems to be something wrong with
the config directory because Flink fails to load the default value
("localhost") from the config. If you had a default value for the job
manager in flink-conf.yaml, it wouldn't fail but only display a wrong
job manager url. Note that it still connects to the right job manager
afterwards.

Sorry for the trouble.

Thanks,
Max

On Tue, Aug 23, 2016 at 11:02 AM, Ufuk Celebi <[hidden email]> wrote:

> You are right that this config key is not needed in this case.
>
> The ClusterClient has been refactored between Flink 1.0 and 1.1 and
> the config parsing might be too strict in this case. It expects the
> IPC address to be set, which is not necessary as you say. It should be
> very easy to fix for 1.1.2. Let's confirm that it is actually a bug
> with Max and file an issue afterwards.
>
> @Max: can you confirm whether this is correct?
>
>
> On Tue, Aug 23, 2016 at 7:24 AM, Hironori Ogibayashi
> <[hidden email]> wrote:
>> Hello,
>>
>> After I upgraded to 1.1.1, I am getting error when submitting job with
>> "flink run"
>>
>> The command and result is like this. It has been working with Flink 1.0.3.
>>
>> ---
>>  % FLINK_CONF_DIR=~/opt/flink/conf ~/opt/flink/flink-1.1.1/bin/flink
>> run -c MyJob target/my-flink-job.jar
>>
>> ------------------------------------------------------------
>>  The program finished with the following exception:
>>
>> java.lang.RuntimeException: Failed to retrieve JobManager address
>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:244)
>>         at org.apache.flink.client.program.StandaloneClusterClient.getClusterIdentifier(StandaloneClusterClient.java:78)
>>         at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:887)
>>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)
>>         at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
>> Caused by: java.lang.IllegalArgumentException: hostname can't be null
>>         at java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
>>         at java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:242)
>>         ... 5 more
>> ---
>>
>> I am using JobManager HA and I set "recovery.mode: zookeeper",
>> recovery.zookeeper.quorum, recovery.zookeeper.path.root is my
>> flink-conf.yaml.
>> So, the client should be able to get JobManager address from zookeeper.
>> If I explicitly specify JobManager address with -m option, it works.
>>
>> Am I missing something?
>>
>> Regards,
>> Hironori Ogibayashi

[Maximilian Michels]

Created an issue and fix should be there soon:
https://issues.apache.org/jira/browse/FLINK-4454

Thanks,
Max

On Tue, Aug 23, 2016 at 4:38 PM, Maximilian Michels <[hidden email]> wrote:

> Hi!
>
> Yes, this is a bug. However, there seems to be something wrong with
> the config directory because Flink fails to load the default value
> ("localhost") from the config. If you had a default value for the job
> manager in flink-conf.yaml, it wouldn't fail but only display a wrong
> job manager url. Note that it still connects to the right job manager
> afterwards.
>
> Sorry for the trouble.
>
> Thanks,
> Max
>
> On Tue, Aug 23, 2016 at 11:02 AM, Ufuk Celebi <[hidden email]> wrote:
>> You are right that this config key is not needed in this case.
>>
>> The ClusterClient has been refactored between Flink 1.0 and 1.1 and
>> the config parsing might be too strict in this case. It expects the
>> IPC address to be set, which is not necessary as you say. It should be
>> very easy to fix for 1.1.2. Let's confirm that it is actually a bug
>> with Max and file an issue afterwards.
>>
>> @Max: can you confirm whether this is correct?
>>
>>
>> On Tue, Aug 23, 2016 at 7:24 AM, Hironori Ogibayashi
>> <[hidden email]> wrote:
>>> Hello,
>>>
>>> After I upgraded to 1.1.1, I am getting error when submitting job with
>>> "flink run"
>>>
>>> The command and result is like this. It has been working with Flink 1.0.3.
>>>
>>> ---
>>>  % FLINK_CONF_DIR=~/opt/flink/conf ~/opt/flink/flink-1.1.1/bin/flink
>>> run -c MyJob target/my-flink-job.jar
>>>
>>> ------------------------------------------------------------
>>>  The program finished with the following exception:
>>>
>>> java.lang.RuntimeException: Failed to retrieve JobManager address
>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:244)
>>>         at org.apache.flink.client.program.StandaloneClusterClient.getClusterIdentifier(StandaloneClusterClient.java:78)
>>>         at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:887)
>>>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)
>>>         at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>>>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
>>> Caused by: java.lang.IllegalArgumentException: hostname can't be null
>>>         at java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
>>>         at java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:242)
>>>         ... 5 more
>>> ---
>>>
>>> I am using JobManager HA and I set "recovery.mode: zookeeper",
>>> recovery.zookeeper.quorum, recovery.zookeeper.path.root is my
>>> flink-conf.yaml.
>>> So, the client should be able to get JobManager address from zookeeper.
>>> If I explicitly specify JobManager address with -m option, it works.
>>>
>>> Am I missing something?
>>>
>>> Regards,
>>> Hironori Ogibayashi

[Hironori Ogibayashi]

Ufuk, Max,

Thank you for your answer and opening JIRA.
I will wait for the fix.

As Max mentioned, I first commented out jobmanager.rpc.address,
jobmanager.rpc.port. When I tried setting localhost and 6123
respectively, it worked.

Regards,
Hironori

2016-08-24 0:54 GMT+09:00 Maximilian Michels <[hidden email]>:

> Created an issue and fix should be there soon:
> https://issues.apache.org/jira/browse/FLINK-4454
>
> Thanks,
> Max
>
> On Tue, Aug 23, 2016 at 4:38 PM, Maximilian Michels <[hidden email]> wrote:
>> Hi!
>>
>> Yes, this is a bug. However, there seems to be something wrong with
>> the config directory because Flink fails to load the default value
>> ("localhost") from the config. If you had a default value for the job
>> manager in flink-conf.yaml, it wouldn't fail but only display a wrong
>> job manager url. Note that it still connects to the right job manager
>> afterwards.
>>
>> Sorry for the trouble.
>>
>> Thanks,
>> Max
>>
>> On Tue, Aug 23, 2016 at 11:02 AM, Ufuk Celebi <[hidden email]> wrote:
>>> You are right that this config key is not needed in this case.
>>>
>>> The ClusterClient has been refactored between Flink 1.0 and 1.1 and
>>> the config parsing might be too strict in this case. It expects the
>>> IPC address to be set, which is not necessary as you say. It should be
>>> very easy to fix for 1.1.2. Let's confirm that it is actually a bug
>>> with Max and file an issue afterwards.
>>>
>>> @Max: can you confirm whether this is correct?
>>>
>>>
>>> On Tue, Aug 23, 2016 at 7:24 AM, Hironori Ogibayashi
>>> <[hidden email]> wrote:
>>>> Hello,
>>>>
>>>> After I upgraded to 1.1.1, I am getting error when submitting job with
>>>> "flink run"
>>>>
>>>> The command and result is like this. It has been working with Flink 1.0.3.
>>>>
>>>> ---
>>>>  % FLINK_CONF_DIR=~/opt/flink/conf ~/opt/flink/flink-1.1.1/bin/flink
>>>> run -c MyJob target/my-flink-job.jar
>>>>
>>>> ------------------------------------------------------------
>>>>  The program finished with the following exception:
>>>>
>>>> java.lang.RuntimeException: Failed to retrieve JobManager address
>>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:244)
>>>>         at org.apache.flink.client.program.StandaloneClusterClient.getClusterIdentifier(StandaloneClusterClient.java:78)
>>>>         at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:887)
>>>>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)
>>>>         at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>>>>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
>>>> Caused by: java.lang.IllegalArgumentException: hostname can't be null
>>>>         at java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
>>>>         at java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
>>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:242)
>>>>         ... 5 more
>>>> ---
>>>>
>>>> I am using JobManager HA and I set "recovery.mode: zookeeper",
>>>> recovery.zookeeper.quorum, recovery.zookeeper.path.root is my
>>>> flink-conf.yaml.
>>>> So, the client should be able to get JobManager address from zookeeper.
>>>> If I explicitly specify JobManager address with -m option, it works.
>>>>
>>>> Am I missing something?
>>>>
>>>> Regards,
>>>> Hironori Ogibayashi

[Maximilian Michels]

Hi Hironori,

That's what I thought. So it won't be an issue for most users who do
not comment out the JobManager url from the config. Still, the
information printed is not correct. The issue has just been fixed.

You will have to wait for the next minor release 1.1.2 or build the
'release-1.1' Git branch.

Best,
Max

On Wed, Aug 24, 2016 at 11:14 AM, Hironori Ogibayashi
<[hidden email]> wrote:

> Ufuk, Max,
>
> Thank you for your answer and opening JIRA.
> I will wait for the fix.
>
> As Max mentioned, I first commented out jobmanager.rpc.address,
> jobmanager.rpc.port. When I tried setting localhost and 6123
> respectively, it worked.
>
> Regards,
> Hironori
>
> 2016-08-24 0:54 GMT+09:00 Maximilian Michels <[hidden email]>:
>> Created an issue and fix should be there soon:
>> https://issues.apache.org/jira/browse/FLINK-4454
>>
>> Thanks,
>> Max
>>
>> On Tue, Aug 23, 2016 at 4:38 PM, Maximilian Michels <[hidden email]> wrote:
>>> Hi!
>>>
>>> Yes, this is a bug. However, there seems to be something wrong with
>>> the config directory because Flink fails to load the default value
>>> ("localhost") from the config. If you had a default value for the job
>>> manager in flink-conf.yaml, it wouldn't fail but only display a wrong
>>> job manager url. Note that it still connects to the right job manager
>>> afterwards.
>>>
>>> Sorry for the trouble.
>>>
>>> Thanks,
>>> Max
>>>
>>> On Tue, Aug 23, 2016 at 11:02 AM, Ufuk Celebi <[hidden email]> wrote:
>>>> You are right that this config key is not needed in this case.
>>>>
>>>> The ClusterClient has been refactored between Flink 1.0 and 1.1 and
>>>> the config parsing might be too strict in this case. It expects the
>>>> IPC address to be set, which is not necessary as you say. It should be
>>>> very easy to fix for 1.1.2. Let's confirm that it is actually a bug
>>>> with Max and file an issue afterwards.
>>>>
>>>> @Max: can you confirm whether this is correct?
>>>>
>>>>
>>>> On Tue, Aug 23, 2016 at 7:24 AM, Hironori Ogibayashi
>>>> <[hidden email]> wrote:
>>>>> Hello,
>>>>>
>>>>> After I upgraded to 1.1.1, I am getting error when submitting job with
>>>>> "flink run"
>>>>>
>>>>> The command and result is like this. It has been working with Flink 1.0.3.
>>>>>
>>>>> ---
>>>>>  % FLINK_CONF_DIR=~/opt/flink/conf ~/opt/flink/flink-1.1.1/bin/flink
>>>>> run -c MyJob target/my-flink-job.jar
>>>>>
>>>>> ------------------------------------------------------------
>>>>>  The program finished with the following exception:
>>>>>
>>>>> java.lang.RuntimeException: Failed to retrieve JobManager address
>>>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:244)
>>>>>         at org.apache.flink.client.program.StandaloneClusterClient.getClusterIdentifier(StandaloneClusterClient.java:78)
>>>>>         at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:887)
>>>>>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)
>>>>>         at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>>>>>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
>>>>> Caused by: java.lang.IllegalArgumentException: hostname can't be null
>>>>>         at java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
>>>>>         at java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
>>>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:242)
>>>>>         ... 5 more
>>>>> ---
>>>>>
>>>>> I am using JobManager HA and I set "recovery.mode: zookeeper",
>>>>> recovery.zookeeper.quorum, recovery.zookeeper.path.root is my
>>>>> flink-conf.yaml.
>>>>> So, the client should be able to get JobManager address from zookeeper.
>>>>> If I explicitly specify JobManager address with -m option, it works.
>>>>>
>>>>> Am I missing something?
>>>>>
>>>>> Regards,
>>>>> Hironori Ogibayashi

[Hironori Ogibayashi]

Max,

Thank you for the fix!

Regards,
Hironori

2016-08-24 18:37 GMT+09:00 Maximilian Michels <[hidden email]>:

> Hi Hironori,
>
> That's what I thought. So it won't be an issue for most users who do
> not comment out the JobManager url from the config. Still, the
> information printed is not correct. The issue has just been fixed.
>
> You will have to wait for the next minor release 1.1.2 or build the
> 'release-1.1' Git branch.
>
> Best,
> Max
>
> On Wed, Aug 24, 2016 at 11:14 AM, Hironori Ogibayashi
> <[hidden email]> wrote:
>> Ufuk, Max,
>>
>> Thank you for your answer and opening JIRA.
>> I will wait for the fix.
>>
>> As Max mentioned, I first commented out jobmanager.rpc.address,
>> jobmanager.rpc.port. When I tried setting localhost and 6123
>> respectively, it worked.
>>
>> Regards,
>> Hironori
>>
>> 2016-08-24 0:54 GMT+09:00 Maximilian Michels <[hidden email]>:
>>> Created an issue and fix should be there soon:
>>> https://issues.apache.org/jira/browse/FLINK-4454
>>>
>>> Thanks,
>>> Max
>>>
>>> On Tue, Aug 23, 2016 at 4:38 PM, Maximilian Michels <[hidden email]> wrote:
>>>> Hi!
>>>>
>>>> Yes, this is a bug. However, there seems to be something wrong with
>>>> the config directory because Flink fails to load the default value
>>>> ("localhost") from the config. If you had a default value for the job
>>>> manager in flink-conf.yaml, it wouldn't fail but only display a wrong
>>>> job manager url. Note that it still connects to the right job manager
>>>> afterwards.
>>>>
>>>> Sorry for the trouble.
>>>>
>>>> Thanks,
>>>> Max
>>>>
>>>> On Tue, Aug 23, 2016 at 11:02 AM, Ufuk Celebi <[hidden email]> wrote:
>>>>> You are right that this config key is not needed in this case.
>>>>>
>>>>> The ClusterClient has been refactored between Flink 1.0 and 1.1 and
>>>>> the config parsing might be too strict in this case. It expects the
>>>>> IPC address to be set, which is not necessary as you say. It should be
>>>>> very easy to fix for 1.1.2. Let's confirm that it is actually a bug
>>>>> with Max and file an issue afterwards.
>>>>>
>>>>> @Max: can you confirm whether this is correct?
>>>>>
>>>>>
>>>>> On Tue, Aug 23, 2016 at 7:24 AM, Hironori Ogibayashi
>>>>> <[hidden email]> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> After I upgraded to 1.1.1, I am getting error when submitting job with
>>>>>> "flink run"
>>>>>>
>>>>>> The command and result is like this. It has been working with Flink 1.0.3.
>>>>>>
>>>>>> ---
>>>>>>  % FLINK_CONF_DIR=~/opt/flink/conf ~/opt/flink/flink-1.1.1/bin/flink
>>>>>> run -c MyJob target/my-flink-job.jar
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>>  The program finished with the following exception:
>>>>>>
>>>>>> java.lang.RuntimeException: Failed to retrieve JobManager address
>>>>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:244)
>>>>>>         at org.apache.flink.client.program.StandaloneClusterClient.getClusterIdentifier(StandaloneClusterClient.java:78)
>>>>>>         at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:887)
>>>>>>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)
>>>>>>         at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)
>>>>>>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
>>>>>> Caused by: java.lang.IllegalArgumentException: hostname can't be null
>>>>>>         at java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
>>>>>>         at java.net.InetSocketAddress.<init>(InetSocketAddress.java:216)
>>>>>>         at org.apache.flink.client.program.ClusterClient.getJobManagerAddressFromConfig(ClusterClient.java:242)
>>>>>>         ... 5 more
>>>>>> ---
>>>>>>
>>>>>> I am using JobManager HA and I set "recovery.mode: zookeeper",
>>>>>> recovery.zookeeper.quorum, recovery.zookeeper.path.root is my
>>>>>> flink-conf.yaml.
>>>>>> So, the client should be able to get JobManager address from zookeeper.
>>>>>> If I explicitly specify JobManager address with -m option, it works.
>>>>>>
>>>>>> Am I missing something?
>>>>>>
>>>>>> Regards,
>>>>>> Hironori Ogibayashi