Service discovery on YARN - find out which port was dynamically assigned to the JobManager Web Interface

[Olivier Solliec]

Hello,

I want to be able to register a flink cluster into a service discovery system (Consul in our case).

This flink cluster is scheduled on YARN.

Is there a way to know which port was assigned to the rest interface ?

Via the rest API /jobmanager/config, I see a key "jobmanager.rpc.address" which is the correct yarn node, but both "rest.port" and "web.port" have "0" value.

The idea is to launch a cluster, retrieve its app id, use the yarn web ui proxy to get the right node address/port, and register this into Consul.

Thank you,

Olivier

[Rong Rong]

As far as I know, the port will be set to random binding. 

Yarn actually have the ability to translate the proxy link to the right node/port. 

If your goal is trying to avoid going through the YARN rest proxy, this could be a problem: There's chances that the host/port will get changed by YARN without a backward notification to Consul (or is there a way I am not sure). This could happen either through YARN's native failure recovery mechanism (2nd attempt) or through HA.

CCed till who might be able to answer more comprehensively regarding this.

Thanks,

Rong

On Wed, Apr 17, 2019 at 7:14 AM Olivier Solliec <[hidden email]> wrote:

Hello,

I want to be able to register a flink cluster into a service discovery system (Consul in our case).

This flink cluster is scheduled on YARN.

Is there a way to know which port was assigned to the rest interface ?

Via the rest API /jobmanager/config, I see a key "jobmanager.rpc.address" which is the correct yarn node, but both "rest.port" and "web.port" have "0" value.

The idea is to launch a cluster, retrieve its app id, use the yarn web ui proxy to get the right node address/port, and register this into Consul.

Thank you,

Olivier

[Till Rohrmann]

Hi Olivier,

since version 1.8 you can set the rest bind port via `rest.bind-port` to a single port or a range. This will now be respected by Yarn deployments. With the next bug fix release of 1.7 you can do the same with `rest.port` but this option only accepts a single port (might lead to port conflicts if you have multiple Flink applications running on the same Yarn cluster with the same configuration).

As Rong pointed out, the port might change if it is not fixed to a single port value in case of container restarts. Thus, you would need to update Consul now and then.

When Flink registers at the Yarn resource manager it sets its rest endpoint as the forwarding destination when using Yarn's proxy. The problem is only that some versions of Yarn only forward GET requests.

Additionally, Flink also registers the rest port at the Yarn resource manager. You can retrieve it by querying the ApplicationReport from the Yarn resource manager and then asking for the `getRpcPort`.

Cheers,

Till

On Thu, Apr 18, 2019 at 3:22 AM Rong Rong <[hidden email]> wrote:

As far as I know, the port will be set to random binding. 

Yarn actually have the ability to translate the proxy link to the right node/port. 

If your goal is trying to avoid going through the YARN rest proxy, this could be a problem: There's chances that the host/port will get changed by YARN without a backward notification to Consul (or is there a way I am not sure). This could happen either through YARN's native failure recovery mechanism (2nd attempt) or through HA.

CCed till who might be able to answer more comprehensively regarding this.

Thanks,

Rong

On Wed, Apr 17, 2019 at 7:14 AM Olivier Solliec <[hidden email]> wrote:

Hello,

I want to be able to register a flink cluster into a service discovery system (Consul in our case).

This flink cluster is scheduled on YARN.

Is there a way to know which port was assigned to the rest interface ?

Via the rest API /jobmanager/config, I see a key "jobmanager.rpc.address" which is the correct yarn node, but both "rest.port" and "web.port" have "0" value.

The idea is to launch a cluster, retrieve its app id, use the yarn web ui proxy to get the right node address/port, and register this into Consul.

Thank you,

Olivier