(DEPRECATED) Apache Flink User Mailing List archive.

REST API "broken" on YARN because POST is not allowed via YARN proxy

Classic

List

Threaded

8 messages Options

Juho Autio

REST API "broken" on YARN because POST is not allowed via YARN proxy

I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL

Cache-Control: must-revalidate,no-cache,no-store

Content-Length: 1523

Content-Type: text/html; charset=iso-8859-1

Date: Wed, 04 Apr 2018 12:48:05 GMT

Pragma: no-cache

<html>

<head>

<title>Error 405 HTTP method POST is not supported by this URL</title>

</head>

<body><h2>HTTP ERROR 405</h2>

Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:

<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty://

</body>

</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found

Access-Control-Allow-Origin: *

Cache-Control: no-cache

Content-Length: 25

Content-Type: application/json; charset=UTF-8

Date: Wed, 04 Apr 2018 12:48:09 GMT

Expires: Wed, 04 Apr 2018 12:48:09 GMT

Pragma: no-cache

{

"errors": [

"Not found."

]

}

^ expected to get this from flink because there's no PUT /checkpoints.

Fabian Hueske-2

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Hi Juho,

Thanks for raising this point!

I'll add Chesnay and Till to the thread who contributed to the REST API.

Best, Fabian

2018-04-04 15:02 GMT+02:00 Juho Autio <[hidden email]>:

I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1523
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 04 Apr 2018 12:48:05 GMT
Date: Wed, 04 Apr 2018 12:48:05 GMT
Pragma: no-cache
Pragma: no-cache

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 405 HTTP method POST is not supported by this URL</title>
</head>
<body><h2>HTTP ERROR 405</h2>
Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty:// 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

</body>
</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 25
Content-Type: application/json; charset=UTF-8
Date: Wed, 04 Apr 2018 12:48:09 GMT
Date: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Pragma: no-cache
Pragma: no-cache

{
"errors": [
"Not found."
]
}

^ expected to get this from flink because there's no PUT /checkpoints.

Till Rohrmann

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Hi Juho,

you are right that due to a limitation in the Yarn proxy [1] we cannot directly contact the cluster through the Yarn proxy.

The way it works at the moment is that the Flink client retrieves the AM's hostname through the ApplicationReport and then directly talks to the AM. This of course requires that one can reach the respective Yarn container. This is a limitation, though not a regression, which we want to improve with the next version of Flink. An idea would be to wrap the REST calls in a GET call to make them pass through the Yarn proxy.

[1] https://issues.apache.org/jira/browse/YARN-2084

Cheers,

Till

On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <[hidden email]> wrote:

Hi Juho,

Thanks for raising this point!

I'll add Chesnay and Till to the thread who contributed to the REST API.

Best, Fabian

2018-04-04 15:02 GMT+02:00 Juho Autio <[hidden email]>:
I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1523
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 04 Apr 2018 12:48:05 GMT
Date: Wed, 04 Apr 2018 12:48:05 GMT
Pragma: no-cache
Pragma: no-cache

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 405 HTTP method POST is not supported by this URL</title>
</head>
<body><h2>HTTP ERROR 405</h2>
Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty:// 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

</body>
</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 25
Content-Type: application/json; charset=UTF-8
Date: Wed, 04 Apr 2018 12:48:09 GMT
Date: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Pragma: no-cache
Pragma: no-cache

{
"errors": [
"Not found."
]
}

^ expected to get this from flink because there's no PUT /checkpoints.

Juho Autio

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Thanks for the answer. Wrapping with GET sounds good to me. You said next version; do you mean that Flink 1.5 would already include this improvement when it's released?

On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <[hidden email]> wrote:

Hi Juho,

you are right that due to a limitation in the Yarn proxy [1] we cannot directly contact the cluster through the Yarn proxy.

The way it works at the moment is that the Flink client retrieves the AM's hostname through the ApplicationReport and then directly talks to the AM. This of course requires that one can reach the respective Yarn container. This is a limitation, though not a regression, which we want to improve with the next version of Flink. An idea would be to wrap the REST calls in a GET call to make them pass through the Yarn proxy.

[1] https://issues.apache.org/jira/browse/YARN-2084

Cheers,
Till

On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <[hidden email]> wrote:
Hi Juho,

Thanks for raising this point!

I'll add Chesnay and Till to the thread who contributed to the REST API.

Best, Fabian

2018-04-04 15:02 GMT+02:00 Juho Autio <[hidden email]>:
I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1523
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 04 Apr 2018 12:48:05 GMT
Date: Wed, 04 Apr 2018 12:48:05 GMT
Pragma: no-cache
Pragma: no-cache

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 405 HTTP method POST is not supported by this URL</title>
</head>
<body><h2>HTTP ERROR 405</h2>
Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty:// 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

</body>
</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 25
Content-Type: application/json; charset=UTF-8
Date: Wed, 04 Apr 2018 12:48:09 GMT
Date: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Pragma: no-cache
Pragma: no-cache

{
"errors": [
"Not found."
]
}

^ expected to get this from flink because there's no PUT /checkpoints.

Till Rohrmann

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

This improvement is unfortunately out of scope for the 1.5 release since the feature freeze is already quite some time ago. But I hope that this improvement will make it into the 1.6 release.

Cheers,

Till

On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <[hidden email]> wrote:

Thanks for the answer. Wrapping with GET sounds good to me. You said next version; do you mean that Flink 1.5 would already include this improvement when it's released?

On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <[hidden email]> wrote:
Hi Juho,

you are right that due to a limitation in the Yarn proxy [1] we cannot directly contact the cluster through the Yarn proxy.

The way it works at the moment is that the Flink client retrieves the AM's hostname through the ApplicationReport and then directly talks to the AM. This of course requires that one can reach the respective Yarn container. This is a limitation, though not a regression, which we want to improve with the next version of Flink. An idea would be to wrap the REST calls in a GET call to make them pass through the Yarn proxy.

[1] https://issues.apache.org/jira/browse/YARN-2084

Cheers,
Till

On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <[hidden email]> wrote:
Hi Juho,

Thanks for raising this point!

I'll add Chesnay and Till to the thread who contributed to the REST API.

Best, Fabian

2018-04-04 15:02 GMT+02:00 Juho Autio <[hidden email]>:
I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1523
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 04 Apr 2018 12:48:05 GMT
Date: Wed, 04 Apr 2018 12:48:05 GMT
Pragma: no-cache
Pragma: no-cache

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 405 HTTP method POST is not supported by this URL</title>
</head>
<body><h2>HTTP ERROR 405</h2>
Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty:// 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

</body>
</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 25
Content-Type: application/json; charset=UTF-8
Date: Wed, 04 Apr 2018 12:48:09 GMT
Date: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Pragma: no-cache
Pragma: no-cache

{
"errors": [
"Not found."
]
}

^ expected to get this from flink because there's no PUT /checkpoints.

Juho Autio

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Hi, I tried to search Flink Jira for this but couldn't find a ticket to match. If there's no ticket yet, did I understand correctly though, that you would be open to support GET as an alternative method for all of Flink's REST operations?

Now that 1.5 was released, it's a pity that this problem kind of prevents us from upgrading – or spend time on creating a workaround.

On Thu, Apr 5, 2018 at 9:10 PM, Till Rohrmann <[hidden email]> wrote:

This improvement is unfortunately out of scope for the 1.5 release since the feature freeze is already quite some time ago. But I hope that this improvement will make it into the 1.6 release.

Cheers,
Till

On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <[hidden email]> wrote:
Thanks for the answer. Wrapping with GET sounds good to me. You said next version; do you mean that Flink 1.5 would already include this improvement when it's released?

On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <[hidden email]> wrote:
Hi Juho,

you are right that due to a limitation in the Yarn proxy [1] we cannot directly contact the cluster through the Yarn proxy.

The way it works at the moment is that the Flink client retrieves the AM's hostname through the ApplicationReport and then directly talks to the AM. This of course requires that one can reach the respective Yarn container. This is a limitation, though not a regression, which we want to improve with the next version of Flink. An idea would be to wrap the REST calls in a GET call to make them pass through the Yarn proxy.

[1] https://issues.apache.org/jira/browse/YARN-2084

Cheers,
Till

On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <[hidden email]> wrote:
Hi Juho,

Thanks for raising this point!

I'll add Chesnay and Till to the thread who contributed to the REST API.

Best, Fabian

2018-04-04 15:02 GMT+02:00 Juho Autio <[hidden email]>:
I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1523
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 04 Apr 2018 12:48:05 GMT
Date: Wed, 04 Apr 2018 12:48:05 GMT
Pragma: no-cache
Pragma: no-cache

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 405 HTTP method POST is not supported by this URL</title>
</head>
<body><h2>HTTP ERROR 405</h2>
Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty:// 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

</body>
</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 25
Content-Type: application/json; charset=UTF-8
Date: Wed, 04 Apr 2018 12:48:09 GMT
Date: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Pragma: no-cache
Pragma: no-cache

{
"errors": [
"Not found."
]
}

^ expected to get this from flink because there's no PUT /checkpoints.

Till Rohrmann

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Hi Juho,

I created a JIRA issue for the problem [1]. Yes, the solution would be to use only GET requests. Either we wrap our requests in a GET request or we change our handlers to accept GET requests only. One thing which we have to fix first is that also the jar file upload goes through REST.

[1] https://issues.apache.org/jira/browse/FLINK-9478

Cheers,

Till

On Wed, May 30, 2018 at 9:07 AM, Juho Autio <[hidden email]> wrote:

Hi, I tried to search Flink Jira for this but couldn't find a ticket to match. If there's no ticket yet, did I understand correctly though, that you would be open to support GET as an alternative method for all of Flink's REST operations?

Now that 1.5 was released, it's a pity that this problem kind of prevents us from upgrading – or spend time on creating a workaround.

On Thu, Apr 5, 2018 at 9:10 PM, Till Rohrmann <[hidden email]> wrote:
This improvement is unfortunately out of scope for the 1.5 release since the feature freeze is already quite some time ago. But I hope that this improvement will make it into the 1.6 release.

Cheers,
Till

On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <[hidden email]> wrote:
Thanks for the answer. Wrapping with GET sounds good to me. You said next version; do you mean that Flink 1.5 would already include this improvement when it's released?

On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <[hidden email]> wrote:
Hi Juho,

you are right that due to a limitation in the Yarn proxy [1] we cannot directly contact the cluster through the Yarn proxy.

The way it works at the moment is that the Flink client retrieves the AM's hostname through the ApplicationReport and then directly talks to the AM. This of course requires that one can reach the respective Yarn container. This is a limitation, though not a regression, which we want to improve with the next version of Flink. An idea would be to wrap the REST calls in a GET call to make them pass through the Yarn proxy.

[1] https://issues.apache.org/jira/browse/YARN-2084

Cheers,
Till

On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <[hidden email]> wrote:
Hi Juho,

Thanks for raising this point!

I'll add Chesnay and Till to the thread who contributed to the REST API.

Best, Fabian

2018-04-04 15:02 GMT+02:00 Juho Autio <[hidden email]>:
I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1523
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 04 Apr 2018 12:48:05 GMT
Date: Wed, 04 Apr 2018 12:48:05 GMT
Pragma: no-cache
Pragma: no-cache

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 405 HTTP method POST is not supported by this URL</title>
</head>
<body><h2>HTTP ERROR 405</h2>
Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty:// 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

</body>
</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 25
Content-Type: application/json; charset=UTF-8
Date: Wed, 04 Apr 2018 12:48:09 GMT
Date: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Pragma: no-cache
Pragma: no-cache

{
"errors": [
"Not found."
]
}

^ expected to get this from flink because there's no PUT /checkpoints.

Juho Autio

Re: REST API "broken" on YARN because POST is not allowed via YARN proxy

Thanks, Till!

On Wed, May 30, 2018 at 10:39 AM, Till Rohrmann <[hidden email]> wrote:

Hi Juho,

I created a JIRA issue for the problem [1]. Yes, the solution would be to use only GET requests. Either we wrap our requests in a GET request or we change our handlers to accept GET requests only. One thing which we have to fix first is that also the jar file upload goes through REST.

[1] https://issues.apache.org/jira/browse/FLINK-9478

Cheers,
Till

On Wed, May 30, 2018 at 9:07 AM, Juho Autio <[hidden email]> wrote:
Hi, I tried to search Flink Jira for this but couldn't find a ticket to match. If there's no ticket yet, did I understand correctly though, that you would be open to support GET as an alternative method for all of Flink's REST operations?

Now that 1.5 was released, it's a pity that this problem kind of prevents us from upgrading – or spend time on creating a workaround.

On Thu, Apr 5, 2018 at 9:10 PM, Till Rohrmann <[hidden email]> wrote:
This improvement is unfortunately out of scope for the 1.5 release since the feature freeze is already quite some time ago. But I hope that this improvement will make it into the 1.6 release.

Cheers,
Till

On Thu, Apr 5, 2018 at 4:45 PM, Juho Autio <[hidden email]> wrote:
Thanks for the answer. Wrapping with GET sounds good to me. You said next version; do you mean that Flink 1.5 would already include this improvement when it's released?

On Thu, Apr 5, 2018 at 2:40 PM, Till Rohrmann <[hidden email]> wrote:
Hi Juho,

you are right that due to a limitation in the Yarn proxy [1] we cannot directly contact the cluster through the Yarn proxy.

The way it works at the moment is that the Flink client retrieves the AM's hostname through the ApplicationReport and then directly talks to the AM. This of course requires that one can reach the respective Yarn container. This is a limitation, though not a regression, which we want to improve with the next version of Flink. An idea would be to wrap the REST calls in a GET call to make them pass through the Yarn proxy.

[1] https://issues.apache.org/jira/browse/YARN-2084

Cheers,
Till

On Wed, Apr 4, 2018 at 4:31 PM, Fabian Hueske <[hidden email]> wrote:
Hi Juho,

Thanks for raising this point!

I'll add Chesnay and Till to the thread who contributed to the REST API.

Best, Fabian

2018-04-04 15:02 GMT+02:00 Juho Autio <[hidden email]>:
I just learned that Flink savepoints API was refactored to require using HTTP POST.

That's fine otherwise, but makes life harder when Flink is run on top of YARN.

I've added example calls below to show how POST is declined by the hadoop-yarn-server-web-proxy*, which only supports GET and PUT.

Can you think of any solution to this? If I would be able to determine the actual host & port for Flink UI, I could use that instead of the proxy address. But that would probably require opening at least one more port, so it's not the optimal solution either. Ideally I would have Flink REST API completely accessible with GET and PUT methods.

To me it seems like AWS EMR will also hit this issue as soon as they start supporting Flink 1.5, because they seem to run Flink as a YARN app.

*) https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java#L296-L306

$ http POST http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 405 HTTP method POST is not supported by this URL
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1523
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 04 Apr 2018 12:48:05 GMT
Date: Wed, 04 Apr 2018 12:48:05 GMT
Pragma: no-cache
Pragma: no-cache

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 405 HTTP method POST is not supported by this URL</title>
</head>
<body><h2>HTTP ERROR 405</h2>
Problem accessing /proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints. Reason:
<pre> HTTP method POST is not supported by this URL</pre><hr />Powered by Jetty:// 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

</body>
</html>

$ http PUT http://10.0.10.71:20888/proxy/application_1522844153347_0001/jobs/652e207f8578574d4a322e23d4f8b908/checkpoints

HTTP/1.1 404 Not Found
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 25
Content-Type: application/json; charset=UTF-8
Date: Wed, 04 Apr 2018 12:48:09 GMT
Date: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Expires: Wed, 04 Apr 2018 12:48:09 GMT
Pragma: no-cache
Pragma: no-cache

{
"errors": [
"Not found."
]
}

^ expected to get this from flink because there's no PUT /checkpoints.