REST rescale with Flink on YARN

[Vasily Melnik]

Hi all.
I'm using Flink 1.8 on YARN with CDH 5.12
When i try to perform rescale request:

curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3'

i get a mistake:
Method PATCH is not defined in RFC 2068 and is not supported by the Servlet API
GET and POST methods work well.
The Server type in response is Jetty(6.1.26.cloudera.4).

How can i deal with this situation?

[Chesnay Schepler]

Older versions of Jetty don't support PATCH requests. You will either have to update it or create a custom Flink version that uses POST for the rescale operation.

On 23/01/2020 13:23, Vasily Melnik wrote:

Hi all.
I'm using Flink 1.8 on YARN with CDH 5.12
When i try to perform rescale request:

curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3'

i get a mistake:
Method PATCH is not defined in RFC 2068 and is not supported by the Servlet API
GET and POST methods work well.
The Server type in response is Jetty(6.1.26.cloudera.4).

How can i deal with this situation?

[Vasily Melnik]

Hi all, 
I've found some solution for this issue. 
Problem is that with YARN ApplicationMaster URL we communicate with JobManager via proxy which is implemented on Jetty 6 (for Hadoop 2.6). 
So to use PATCH method we need to locate original JobManager URL. 
Using /jobmanager/config API we could get only host, but web.port is displayed as 0 (???)
To find actual web port, we should parse YARN logs for jobmanager, where we can find something like this:

INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at <host>:<port>.

Maybe  someone knows less complicated way to find actual REST URL under YARN?
  

On Thu, 23 Jan 2020 at 15:32, Chesnay Schepler <[hidden email]> wrote:

Older versions of Jetty don't support PATCH requests. You will either have to update it or create a custom Flink version that uses POST for the rescale operation.

On 23/01/2020 13:23, Vasily Melnik wrote:

Hi all.
I'm using Flink 1.8 on YARN with CDH 5.12
When i try to perform rescale request:

curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3'

i get a mistake:
Method PATCH is not defined in RFC 2068 and is not supported by the Servlet API
GET and POST methods work well.
The Server type in response is Jetty(6.1.26.cloudera.4).

How can i deal with this situation?

С уважением,
Василий Мельник

[Gary Yao-4]

Hi,

You can use

    yarn application -status <APPLICATION_ID>

to find the host and port that the server is listening on (AM host & RPC
Port). If you need to access that information programmatically, take a look at
the YarnClient [1].

Best,
Gary


[1] https://hadoop.apache.org/docs/r2.8.5/api/org/apache/hadoop/yarn/client/api/YarnClient.html

On Thu, Jan 23, 2020 at 3:21 PM Vasily Melnik <[hidden email]> wrote:

Hi all, 
I've found some solution for this issue. 
Problem is that with YARN ApplicationMaster URL we communicate with JobManager via proxy which is implemented on Jetty 6 (for Hadoop 2.6). 
So to use PATCH method we need to locate original JobManager URL. 
Using /jobmanager/config API we could get only host, but web.port is displayed as 0 (???)
To find actual web port, we should parse YARN logs for jobmanager, where we can find something like this:

INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at <host>:<port>.

Maybe  someone knows less complicated way to find actual REST URL under YARN?
  

С уважением,
Василий Мельник

On Thu, 23 Jan 2020 at 15:32, Chesnay Schepler <[hidden email]> wrote:

Older versions of Jetty don't support PATCH requests. You will either have to update it or create a custom Flink version that uses POST for the rescale operation.

On 23/01/2020 13:23, Vasily Melnik wrote:

Hi all.
I'm using Flink 1.8 on YARN with CDH 5.12
When i try to perform rescale request:

curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3'

i get a mistake:
Method PATCH is not defined in RFC 2068 and is not supported by the Servlet API
GET and POST methods work well.
The Server type in response is Jetty(6.1.26.cloudera.4).

How can i deal with this situation?

С уважением,
Василий Мельник

[Yang Wang]

Gary is right. You could also access the Yarn RM rest api to get the AM original address.

http://hadoop.apache.org/docs/r2.8.5/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_API

Best,

Yang

Gary Yao <[hidden email]> 于2020年1月28日周二 下午6:17写道：

Hi,

You can use

    yarn application -status <APPLICATION_ID>

to find the host and port that the server is listening on (AM host & RPC
Port). If you need to access that information programmatically, take a look at
the YarnClient [1].

Best,
Gary


[1] https://hadoop.apache.org/docs/r2.8.5/api/org/apache/hadoop/yarn/client/api/YarnClient.html

On Thu, Jan 23, 2020 at 3:21 PM Vasily Melnik <[hidden email]> wrote:

Hi all, 
I've found some solution for this issue. 
Problem is that with YARN ApplicationMaster URL we communicate with JobManager via proxy which is implemented on Jetty 6 (for Hadoop 2.6). 
So to use PATCH method we need to locate original JobManager URL. 
Using /jobmanager/config API we could get only host, but web.port is displayed as 0 (???)
To find actual web port, we should parse YARN logs for jobmanager, where we can find something like this:

INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint - Rest endpoint listening at <host>:<port>.

Maybe  someone knows less complicated way to find actual REST URL under YARN?
  

С уважением,
Василий Мельник

On Thu, 23 Jan 2020 at 15:32, Chesnay Schepler <[hidden email]> wrote:

Older versions of Jetty don't support PATCH requests. You will either have to update it or create a custom Flink version that uses POST for the rescale operation.

On 23/01/2020 13:23, Vasily Melnik wrote:

Hi all.
I'm using Flink 1.8 on YARN with CDH 5.12
When i try to perform rescale request:

curl -v -X PATCH '<my_url>/proxy/application_1576854986116_0079/jobs/11dcfc3163936fc019e049fc841b075b/rescaling?parallelism=3'

i get a mistake:
Method PATCH is not defined in RFC 2068 and is not supported by the Servlet API
GET and POST methods work well.
The Server type in response is Jetty(6.1.26.cloudera.4).

How can i deal with this situation?

С уважением,
Василий Мельник