Trace jar file name from jobId, is that possible?

[Hao Sun]

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.

[Fabian Hueske-2]

Hi,

you can submit jar files and start jobs via the REST interface [1].

When starting a job, you get the jobId. You can link jar files and savepoints via the jobId.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs

2017-12-02 0:28 GMT+01:00 Hao Sun <[hidden email]>:

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.

[Hao Sun]

Thanks Fabian, there is one case can not be covered by the REST API. When a job rescheduled to run, but jobid will change, and I wont be able to backtrace the jar name. Why not keep the jar name stored somewhere and expose it through the api as well?

On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <[hidden email]> wrote:

Hi,

you can submit jar files and start jobs via the REST interface [1].

When starting a job, you get the jobId. You can link jar files and savepoints via the jobId.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs

2017-12-02 0:28 GMT+01:00 Hao Sun <[hidden email]>:

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.

[Hao Sun]

Anything I can do for the job reschedule case? Thanks.
Or is there a way to add job lifecycle hooks to trace it?

On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <[hidden email]> wrote:

Thanks Fabian, there is one case can not be covered by the REST API. When a job rescheduled to run, but jobid will change, and I wont be able to backtrace the jar name. Why not keep the jar name stored somewhere and expose it through the api as well?

On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <[hidden email]> wrote:

Hi,

you can submit jar files and start jobs via the REST interface [1].

When starting a job, you get the jobId. You can link jar files and savepoints via the jobId.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs

2017-12-02 0:28 GMT+01:00 Hao Sun <[hidden email]>:

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.

[Fabian Hueske-2]

What do you mean by rescheduled?
Started from a savepoint or restarted during failure recovery?

2017-12-07 16:59 GMT+01:00 Hao Sun <[hidden email]>:

Anything I can do for the job reschedule case? Thanks.
Or is there a way to add job lifecycle hooks to trace it?

On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <[hidden email]> wrote:

Thanks Fabian, there is one case can not be covered by the REST API. When a job rescheduled to run, but jobid will change, and I wont be able to backtrace the jar name. Why not keep the jar name stored somewhere and expose it through the api as well?

On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <[hidden email]> wrote:

Hi,

you can submit jar files and start jobs via the REST interface [1].

When starting a job, you get the jobId. You can link jar files and savepoints via the jobId.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs

2017-12-02 0:28 GMT+01:00 Hao Sun <[hidden email]>:

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.

[Hao Sun]

I mean restarted during failure recovery

On Thu, Dec 7, 2017 at 8:29 AM Fabian Hueske <[hidden email]> wrote:

What do you mean by rescheduled?
Started from a savepoint or restarted during failure recovery?

2017-12-07 16:59 GMT+01:00 Hao Sun <[hidden email]>:

Anything I can do for the job reschedule case? Thanks.
Or is there a way to add job lifecycle hooks to trace it?

On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <[hidden email]> wrote:

Thanks Fabian, there is one case can not be covered by the REST API. When a job rescheduled to run, but jobid will change, and I wont be able to backtrace the jar name. Why not keep the jar name stored somewhere and expose it through the api as well?

On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <[hidden email]> wrote:

Hi,

you can submit jar files and start jobs via the REST interface [1].

When starting a job, you get the jobId. You can link jar files and savepoints via the jobId.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs

2017-12-02 0:28 GMT+01:00 Hao Sun <[hidden email]>:

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.

[Fabian Hueske-2]

AFAIK, a job keeps its ID in case of a recovery.

Did you observe something else?

2017-12-07 17:32 GMT+01:00 Hao Sun <[hidden email]>:

I mean restarted during failure recovery

On Thu, Dec 7, 2017 at 8:29 AM Fabian Hueske <[hidden email]> wrote:

What do you mean by rescheduled?
Started from a savepoint or restarted during failure recovery?

2017-12-07 16:59 GMT+01:00 Hao Sun <[hidden email]>:

Anything I can do for the job reschedule case? Thanks.
Or is there a way to add job lifecycle hooks to trace it?

On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <[hidden email]> wrote:

Thanks Fabian, there is one case can not be covered by the REST API. When a job rescheduled to run, but jobid will change, and I wont be able to backtrace the jar name. Why not keep the jar name stored somewhere and expose it through the api as well?

On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <[hidden email]> wrote:

Hi,

you can submit jar files and start jobs via the REST interface [1].

When starting a job, you get the jobId. You can link jar files and savepoints via the jobId.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs

2017-12-02 0:28 GMT+01:00 Hao Sun <[hidden email]>:

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.

[Hao Sun]

Let me check details, on top of my mind I remember the job id changes, I might be wrong.

On Thu, Dec 7, 2017, 08:48 Fabian Hueske <[hidden email]> wrote:

AFAIK, a job keeps its ID in case of a recovery.

Did you observe something else?

2017-12-07 17:32 GMT+01:00 Hao Sun <[hidden email]>:

I mean restarted during failure recovery

On Thu, Dec 7, 2017 at 8:29 AM Fabian Hueske <[hidden email]> wrote:

What do you mean by rescheduled?
Started from a savepoint or restarted during failure recovery?

2017-12-07 16:59 GMT+01:00 Hao Sun <[hidden email]>:

Anything I can do for the job reschedule case? Thanks.
Or is there a way to add job lifecycle hooks to trace it?

On Mon, Dec 4, 2017 at 12:01 PM Hao Sun <[hidden email]> wrote:

Thanks Fabian, there is one case can not be covered by the REST API. When a job rescheduled to run, but jobid will change, and I wont be able to backtrace the jar name. Why not keep the jar name stored somewhere and expose it through the api as well?

On Mon, Dec 4, 2017 at 4:52 AM Fabian Hueske <[hidden email]> wrote:

Hi,

you can submit jar files and start jobs via the REST interface [1].

When starting a job, you get the jobId. You can link jar files and savepoints via the jobId.

Best, Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html#submitting-programs

2017-12-02 0:28 GMT+01:00 Hao Sun <[hidden email]>:

Hi I am using Flink 1.3.2 on K8S, and need a deployment strategy for my app.

I want to use savepoints to resume a job after each deployment.

As you know I need jar file name and path to savepoints to resume a task.

Currently `flink list` command only gives me job ids, not jar file names.

And REST API does not have that information as well. If I have multiple jar files how can I map the savepoints back to jars, so I can resume my task?

I thought about save the jar to jid map somewhere, but Flink can reschedule a task on failures, so the map will be stale.

Any thoughts is appreciated. Many thanks.