1.1.1: JobManager config endpoint no longer supplies port

[Shannon Carey]

It appears that when running Flink 1.1.1 on Yarn, my previous method of making a request to the yarn AM proxy on the master node at <a href="http://{master_node}:20888/proxy/{app_id}/jobmanager/config">http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the same as it did.

Previously, the returned JSON value would include an accurate value for "jobmanager.web.port". Now, however, it appears to return a zero.

As a result, I cannot expose the JobManager UI itself. I can use the Yarn proxy but it does not support the useful feature of uploading JARs.

Is there another way of determining which port the JobManager is running on? Thanks!

-Shannon

[Ufuk Celebi]

Hey Shannon! I just took a look at the code and it looks like the
Flink REST handler for the config did _not_ change since last year. It
could be that somehow the config is loaded differently. Can you verify
that using the same config with Flink 1.0 and Flink 1.1 the port is
printed correctly and as 0? I will look into it as well.

The JAR uploading issue is a separate one, correct?

On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey <[hidden email]> wrote:

> It appears that when running Flink 1.1.1 on Yarn, my previous method of
> making a request to the yarn AM proxy on the master node at
> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the
> same as it did.
>
> Previously, the returned JSON value would include an accurate value for
> "jobmanager.web.port". Now, however, it appears to return a zero.
>
> As a result, I cannot expose the JobManager UI itself. I can use the Yarn
> proxy but it does not support the useful feature of uploading JARs.
>
> Is there another way of determining which port the JobManager is running on?
> Thanks!
>
> -Shannon

[Ufuk Celebi]

I've verified this. I think this has likely accidentally changed with
the refactoring of the YARN setup for Flink 1.1. We probably wrote the
web monitor port explicitly to the config in 1.0 whereas we don't do
it in 1.1 anymore. I think this should be addressed with the next
bugfix release 1.1.2.

What comes to mind to find out the port is to parse the logs, but I
don't think that's feasible, is it? Would also be interested what you
are referring to with the JARs.

On Mon, Aug 15, 2016 at 4:04 PM, Ufuk Celebi <[hidden email]> wrote:

> Hey Shannon! I just took a look at the code and it looks like the
> Flink REST handler for the config did _not_ change since last year. It
> could be that somehow the config is loaded differently. Can you verify
> that using the same config with Flink 1.0 and Flink 1.1 the port is
> printed correctly and as 0? I will look into it as well.
>
> The JAR uploading issue is a separate one, correct?
>
> On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey <[hidden email]> wrote:
>> It appears that when running Flink 1.1.1 on Yarn, my previous method of
>> making a request to the yarn AM proxy on the master node at
>> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the
>> same as it did.
>>
>> Previously, the returned JSON value would include an accurate value for
>> "jobmanager.web.port". Now, however, it appears to return a zero.
>>
>> As a result, I cannot expose the JobManager UI itself. I can use the Yarn
>> proxy but it does not support the useful feature of uploading JARs.
>>
>> Is there another way of determining which port the JobManager is running on?
>> Thanks!
>>
>> -Shannon

[Shannon Carey]

Thanks Ufuk. For now, we will use the Yarn AM proxy.

About uploading JARs: the JobManager UI that is exposed via the Yarn AM proxy does not allow manually uploading Flink job jars for execution on the cluster (look for "Yarn's AM proxy doesn't allow file uploads." in the code). As I understand it, this is due choices Yarn has made about security.



On 8/15/16, 9:25 AM, "Ufuk Celebi" <[hidden email]> wrote:

>I've verified this. I think this has likely accidentally changed with
>the refactoring of the YARN setup for Flink 1.1. We probably wrote the
>web monitor port explicitly to the config in 1.0 whereas we don't do
>it in 1.1 anymore. I think this should be addressed with the next
>bugfix release 1.1.2.
>
>What comes to mind to find out the port is to parse the logs, but I
>don't think that's feasible, is it? Would also be interested what you
>are referring to with the JARs.
>
>On Mon, Aug 15, 2016 at 4:04 PM, Ufuk Celebi <[hidden email]> wrote:
>> Hey Shannon! I just took a look at the code and it looks like the
>> Flink REST handler for the config did _not_ change since last year. It
>> could be that somehow the config is loaded differently. Can you verify
>> that using the same config with Flink 1.0 and Flink 1.1 the port is
>> printed correctly and as 0? I will look into it as well.
>>
>> The JAR uploading issue is a separate one, correct?
>>
>> On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey <[hidden email]> wrote:
>>> It appears that when running Flink 1.1.1 on Yarn, my previous method of
>>> making a request to the yarn AM proxy on the master node at
>>> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the
>>> same as it did.
>>>
>>> Previously, the returned JSON value would include an accurate value for
>>> "jobmanager.web.port". Now, however, it appears to return a zero.
>>>
>>> As a result, I cannot expose the JobManager UI itself. I can use the Yarn
>>> proxy but it does not support the useful feature of uploading JARs.
>>>
>>> Is there another way of determining which port the JobManager is running on?
>>> Thanks!
>>>
>>> -Shannon
>

[rmetzger0]

Hi Shannon,

the problem is that YARNs proxy only allows GET HTTP requests, but for uploading files, a different request type is needed.

I've filed a JIRA for the problem you've reported: https://issues.apache.org/jira/browse/FLINK-4432

Regards,

Robert

On Mon, Aug 15, 2016 at 6:03 PM, Shannon Carey <[hidden email]> wrote:

Thanks Ufuk. For now, we will use the Yarn AM proxy.

About uploading JARs: the JobManager UI that is exposed via the Yarn AM proxy does not allow manually uploading Flink job jars for execution on the cluster (look for "Yarn's AM proxy doesn't allow file uploads." in the code). As I understand it, this is due choices Yarn has made about security.

On 8/15/16, 9:25 AM, "Ufuk Celebi" <[hidden email]> wrote:

>I've verified this. I think this has likely accidentally changed with
>the refactoring of the YARN setup for Flink 1.1. We probably wrote the
>web monitor port explicitly to the config in 1.0 whereas we don't do
>it in 1.1 anymore. I think this should be addressed with the next
>bugfix release 1.1.2.
>
>What comes to mind to find out the port is to parse the logs, but I
>don't think that's feasible, is it? Would also be interested what you
>are referring to with the JARs.
>
>On Mon, Aug 15, 2016 at 4:04 PM, Ufuk Celebi <[hidden email]> wrote:
>> Hey Shannon! I just took a look at the code and it looks like the
>> Flink REST handler for the config did _not_ change since last year. It
>> could be that somehow the config is loaded differently. Can you verify
>> that using the same config with Flink 1.0 and Flink 1.1 the port is
>> printed correctly and as 0? I will look into it as well.
>>
>> The JAR uploading issue is a separate one, correct?
>>
>> On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey <[hidden email]> wrote:
>>> It appears that when running Flink 1.1.1 on Yarn, my previous method of
>>> making a request to the yarn AM proxy on the master node at
>>> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the
>>> same as it did.
>>>
>>> Previously, the returned JSON value would include an accurate value for
>>> "jobmanager.web.port". Now, however, it appears to return a zero.
>>>
>>> As a result, I cannot expose the JobManager UI itself. I can use the Yarn
>>> proxy but it does not support the useful feature of uploading JARs.
>>>
>>> Is there another way of determining which port the JobManager is running on?
>>> Thanks!
>>>
>>> -Shannon
>

[Shannon Carey]

A followup (in case anyone is interested): we worked around this by making a request to the "/jars" endpoint of the UI. The response has an attribute called "address" which includes the DNS name and port where the UI is accessible.