(DEPRECATED) Apache Flink User Mailing List archive.

How can I submit a flink job to YARN/Cluster from java code?

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
程骥
Reply | Threaded
Open this post in threaded view
♦
♦
|

How can I submit a flink job to YARN/Cluster from java code?

程骥
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible? 
Is there anyone tried to do it before ？ 


Thanks
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: How can I submit a flink job to YARN/Cluster from java code?

Till Rohrmann

Hi,

it is possible to submit a job to a running cluster (Standalone/Mesos/Yarn) by using the RemoteEnvironment which you can create via ExecutionEnvironment.createRemoteEnvironment and then executing your program (e.g. running it in the IDE). This assumes that the remote cluster is already running.

The RemoteEnvironment requires that you give it the hostname and port of the running cluster or, if using a HA cluster, a properly configured configuration from which the client can retrieve the HA settings.

Cheers,
Till

​

On Fri, Jul 28, 2017 at 9:50 AM, 程骥 <[hidden email]> wrote:
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible? 
Is there anyone tried to do it before ？ 


Thanks
zrc@zjdex.com
Reply | Threaded
Open this post in threaded view
♦
♦
|

re: How can I submit a flink job to YARN/Cluster from java code?

zrc@zjdex.com
In reply to this post by 程骥
Hi:
    I think you can use Flink restful API, like  "POST: /jars/MyProgram.jar/run?savepointPath=/my-savepoints/savepoint-1bae02a80464&allowNonRestoredState=true"
   The detail information of restful API , you can see the link https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html .

 
发件人： [hidden email]
发送时间： 2017-07-28 15:50
收件人： [hidden email]
主题： How can I submit a flink job to YARN/Cluster from java code?
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible? 
Is there anyone tried to do it before ？ 


Thanks
程骥
Reply | Threaded
Open this post in threaded view
♦
♦
|

回复： How can I submit a flink job to YARN/Cluster from java code?

程骥
In reply to this post by Till Rohrmann
Hi,Till
Actually,to submit a flink job to flink clusetr with ExecutionEnvironment.createRemoteEnvironment,I have implemented.

but how to submit flink job to flink on yarn,I have no idea.

I think this API not supported.

Thanks for your help.
:)

------------------ 原始邮件 ------------------
发件人: "Till Rohrmann";<[hidden email]>;
发送时间: 2017年7月28日(星期五) 下午5:13
收件人: "程骥"<[hidden email]>;
抄送: "user"<[hidden email]>;
主题: Re: How can I submit a flink job to YARN/Cluster from java code?

Hi,

it is possible to submit a job to a running cluster (Standalone/Mesos/Yarn) by using the RemoteEnvironment which you can create via ExecutionEnvironment.createRemoteEnvironment and then executing your program (e.g. running it in the IDE). This assumes that the remote cluster is already running.

The RemoteEnvironment requires that you give it the hostname and port of the running cluster or, if using a HA cluster, a properly configured configuration from which the client can retrieve the HA settings.

Cheers,
Till

​

On Fri, Jul 28, 2017 at 9:50 AM, 程骥 <[hidden email]> wrote:
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible? 
Is there anyone tried to do it before ？ 


Thanks
程骥
Reply | Threaded
Open this post in threaded view
♦
♦
|

回复：re: How can I submit a flink job to YARN/Cluster from java code?

程骥
In reply to this post by zrc@zjdex.com
Hi,
Thanks for yor help.
This is a good idea.
Maybe I didn't understand the information of restful API.
I think the restful API  can not submit a job to yarn.
Is that right?



------------------ 原始邮件 ------------------
发送时间: 2017年7月28日(星期五) 下午5:15
收件人: "程骥"<[hidden email]>; "user"<[hidden email]>;
主题: re: How can I submit a flink job to YARN/Cluster from java code?

Hi:
    I think you can use Flink restful API, like  "POST: /jars/MyProgram.jar/run?savepointPath=/my-savepoints/savepoint-1bae02a80464&allowNonRestoredState=true"
   The detail information of restful API , you can see the link https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html .

 
发件人： [hidden email]
发送时间： 2017-07-28 15:50
收件人： [hidden email]
主题： How can I submit a flink job to YARN/Cluster from java code?
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible?
Is there anyone tried to do it before ？ 


Thanks
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: re: How can I submit a flink job to YARN/Cluster from java code?

Till Rohrmann

The RemoteEnvironment won’t allow you to spawn a Flink cluster running on Yarn. But you can start one with yarn-session.sh. The same applies to the REST endpoint. You first have to start a cluster.

Cheers,
Till

​

On Fri, Jul 28, 2017 at 12:14 PM, 程骥 <[hidden email]> wrote:
Hi,
Thanks for yor help.
This is a good idea.
Maybe I didn't understand the information of restful API.
I think the restful API  can not submit a job to yarn.
Is that right?



------------------ 原始邮件 ------------------
发送时间: 2017年7月28日(星期五) 下午5:15
收件人: "程骥"<[hidden email]>; "user"<[hidden email]>;
主题: re: How can I submit a flink job to YARN/Cluster from java code?

Hi:
    I think you can use Flink restful API, like  "POST: /jars/MyProgram.jar/run?savepointPath=/my-savepoints/savepoint-1bae02a80464&allowNonRestoredState=true"
   The detail information of restful API , you can see the link https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html .

 
发件人： [hidden email]
发送时间： 2017-07-28 15:50
收件人： [hidden email]
主题： How can I submit a flink job to YARN/Cluster from java code?
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible?
Is there anyone tried to do it before ？ 


Thanks
zrc@zjdex.com
Reply | Threaded
Open this post in threaded view
♦
♦
|

re: How can I submit a flink job to YARN/Cluster from java code?

zrc@zjdex.com
In reply to this post by 程骥
 In this link, it is not said the restful API can not used in yarn cluster. I think it is ok, and you can try. :)


 
发件人： [hidden email]
发送时间： 2017-07-28 18:14
主题： 回复：re: How can I submit a flink job to YARN/Cluster from java code?
Hi,
Thanks for yor help.
This is a good idea.
Maybe I didn't understand the information of restful API.
I think the restful API  can not submit a job to yarn.
Is that right?



------------------ 原始邮件 ------------------
发送时间: 2017年7月28日(星期五) 下午5:15
收件人: "程骥"<[hidden email]>; "user"<[hidden email]>;
主题: re: How can I submit a flink job to YARN/Cluster from java code?

Hi:
    I think you can use Flink restful API, like  "POST: /jars/MyProgram.jar/run?savepointPath=/my-savepoints/savepoint-1bae02a80464&allowNonRestoredState=true"
   The detail information of restful API , you can see the link https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html .

 
发件人： [hidden email]
发送时间： 2017-07-28 15:50
收件人： [hidden email]
主题： How can I submit a flink job to YARN/Cluster from java code?
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible?
Is there anyone tried to do it before ？ 


Thanks
程骥
Reply | Threaded
Open this post in threaded view
♦
♦
|

回复：re: How can I submit a flink job to YARN/Cluster from java code?

程骥
ok,thank you.


------------------ 原始邮件 ------------------
发送时间: 2017年7月31日(星期一) 上午9:31
收件人: "程骥"<[hidden email]>; "user"<[hidden email]>;
主题: re: How can I submit a flink job to YARN/Cluster from java code?

 In this link, it is not said the restful API can not used in yarn cluster. I think it is ok, and you can try. :)


 
发件人： [hidden email]
发送时间： 2017-07-28 18:14
主题： 回复：re: How can I submit a flink job to YARN/Cluster from java code?
Hi,
Thanks for yor help.
This is a good idea.
Maybe I didn't understand the information of restful API.
I think the restful API  can not submit a job to yarn.
Is that right?



------------------ 原始邮件 ------------------
发送时间: 2017年7月28日(星期五) 下午5:15
收件人: "程骥"<[hidden email]>; "user"<[hidden email]>;
主题: re: How can I submit a flink job to YARN/Cluster from java code?

Hi:
    I think you can use Flink restful API, like  "POST: /jars/MyProgram.jar/run?savepointPath=/my-savepoints/savepoint-1bae02a80464&allowNonRestoredState=true"
   The detail information of restful API , you can see the link https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html .

 
发件人： [hidden email]
发送时间： 2017-07-28 15:50
收件人： [hidden email]
主题： How can I submit a flink job to YARN/Cluster from java code?
Hello，
I want to submit a flink job to YARN/Cluster from java code.If this is feasible?
Is there anyone tried to do it before ？ 


Thanks
Free forum by Nabble Edit this page