Retrieving Flink job ID/YARN Id programmatically

[Lei Nie]

Hello,

I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).

Thanks

[vino yang]

Hi Lei Nie,

You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.

Best,

Vino

Lei Nie <[hidden email]> 于2019年11月8日周五 上午8:38写道：

Hello,

I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).

Thanks

[Lei Nie]

This does not get the correct id:
StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
eea5abc21dd8743a4090f4a3a660f9e8
Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a



On Thu, Nov 7, 2019 at 6:56 PM vino yang <[hidden email]> wrote:

>
> Hi Lei Nie,
>
> You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.
>
> Best,
> Vino
>
> Lei Nie <[hidden email]> 于2019年11月8日周五 上午8:38写道：
>>
>> Hello,
>> I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).
>>
>> Thanks

[Lei Nie]

I looked at the code, and
StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
generating a random ID unrelated to the actual ID used.

Is there any way to fetch the real ID at runtime?
Use case: fetch most recent checkpoint from stable storage for
automated restarts. Most recent checkpoint has form
".../checkpoints/flink_app_id/chk-123"

On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <[hidden email]> wrote:

>
> This does not get the correct id:
> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
> eea5abc21dd8743a4090f4a3a660f9e8
> Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
>
>
>
> On Thu, Nov 7, 2019 at 6:56 PM vino yang <[hidden email]> wrote:
> >
> > Hi Lei Nie,
> >
> > You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.
> >
> > Best,
> > Vino
> >
> > Lei Nie <[hidden email]> 于2019年11月8日周五 上午8:38写道：
> >>
> >> Hello,
> >> I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).
> >>
> >> Thanks

[vino yang]

Hi Lei,

It would be better to use Flink's RESTful API to fetch the information of the running jobs[1].

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1

Best,

Vino

Lei Nie <[hidden email]> 于2019年11月22日周五 上午4:14写道：

I looked at the code, and
StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
generating a random ID unrelated to the actual ID used.

Is there any way to fetch the real ID at runtime?
Use case: fetch most recent checkpoint from stable storage for
automated restarts. Most recent checkpoint has form
".../checkpoints/flink_app_id/chk-123"

On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <[hidden email]> wrote:
>
> This does not get the correct id:
> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
> eea5abc21dd8743a4090f4a3a660f9e8
> Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
>
>
>
> On Thu, Nov 7, 2019 at 6:56 PM vino yang <[hidden email]> wrote:
> >
> > Hi Lei Nie,
> >
> > You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.
> >
> > Best,
> > Vino
> >
> > Lei Nie <[hidden email]> 于2019年11月8日周五 上午8:38写道：
> >>
> >> Hello,
> >> I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).
> >>
> >> Thanks

[Ana]

Hi Lei,

To add, you may use Hadoop Resource Manager REST APIs https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html. I'm also running Flink application on YARN and use this API for that purpose. If you find other way or a much better solution, please let me know!

Regards,

Ana

On Fri, Nov 22, 2019 at 10:58 AM vino yang <[hidden email]> wrote:

Hi Lei,

It would be better to use Flink's RESTful API to fetch the information of the running jobs[1].

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1

Best,

Vino

Lei Nie <[hidden email]> 于2019年11月22日周五 上午4:14写道：

I looked at the code, and
StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
generating a random ID unrelated to the actual ID used.

Is there any way to fetch the real ID at runtime?
Use case: fetch most recent checkpoint from stable storage for
automated restarts. Most recent checkpoint has form
".../checkpoints/flink_app_id/chk-123"

On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <[hidden email]> wrote:
>
> This does not get the correct id:
> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
> eea5abc21dd8743a4090f4a3a660f9e8
> Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
>
>
>
> On Thu, Nov 7, 2019 at 6:56 PM vino yang <[hidden email]> wrote:
> >
> > Hi Lei Nie,
> >
> > You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.
> >
> > Best,
> > Vino
> >
> > Lei Nie <[hidden email]> 于2019年11月8日周五 上午8:38写道：
> >>
> >> Hello,
> >> I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).
> >>
> >> Thanks

[Arvid Heise-3]

Hi Lei,

if you use

public JobExecutionResult StreamExecutionEnvironment#execute()

You can retrieve the job id through the result.

result.getJobID()

Best,

Arvid

On Mon, Nov 25, 2019 at 3:50 AM Ana <[hidden email]> wrote:

Hi Lei,

To add, you may use Hadoop Resource Manager REST APIs https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html. I'm also running Flink application on YARN and use this API for that purpose. If you find other way or a much better solution, please let me know!

Regards,

Ana

On Fri, Nov 22, 2019 at 10:58 AM vino yang <[hidden email]> wrote:

Hi Lei,

It would be better to use Flink's RESTful API to fetch the information of the running jobs[1].

[1]: https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html#jobs-1

Best,

Vino

Lei Nie <[hidden email]> 于2019年11月22日周五 上午4:14写道：

I looked at the code, and
StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID is
generating a random ID unrelated to the actual ID used.

Is there any way to fetch the real ID at runtime?
Use case: fetch most recent checkpoint from stable storage for
automated restarts. Most recent checkpoint has form
".../checkpoints/flink_app_id/chk-123"

On Thu, Nov 21, 2019 at 11:28 AM Lei Nie <[hidden email]> wrote:
>
> This does not get the correct id:
> StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID =
> eea5abc21dd8743a4090f4a3a660f9e8
> Actual job ID (from webUI): 1357d21be640b6a3b8a86a063f4bba8a
>
>
>
> On Thu, Nov 7, 2019 at 6:56 PM vino yang <[hidden email]> wrote:
> >
> > Hi Lei Nie,
> >
> > You can use `StreamExecutionEnvironment#getStreamGraph#getJobGraph#getJobID` to get the job id.
> >
> > Best,
> > Vino
> >
> > Lei Nie <[hidden email]> 于2019年11月8日周五 上午8:38写道：
> >>
> >> Hello,
> >> I am currently executing streaming jobs via StreamExecutionEnvironment. Is it possible to retrieve the Flink job ID/YARN ID within the context of a job? I'd like to be able to automatically register the job such that monitoring jobs can run (REST api requires for example job id).
> >>
> >> Thanks