Savepoint Location from Flink REST API

[Aaron Langford]

Hey Flink Community,

I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be available via the REST API. Does this exist today?

Aaron

[r_khachatryan]

Hey Aaron,

You can use /jobs/:jobid/savepoints/:triggerid to get the location when the checkpoint is completed.

Please see https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html

Meanwhile, I've created an issue to update the docs: https://issues.apache.org/jira/browse/FLINK-16696

On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <[hidden email]> wrote:

Hey Flink Community,

I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be available via the REST API. Does this exist today?

Aaron

[Aaron Langford]

Roman,

Thanks for the info. That's super helpful. I'd be interested in picking that ticket up.

One additional question: the states that can return from this API are only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures represented for this endpoint?

Aaron

On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <[hidden email]> wrote:

Hey Aaron,

You can use /jobs/:jobid/savepoints/:triggerid to get the location when the checkpoint is completed.

Please see https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html

Meanwhile, I've created an issue to update the docs: https://issues.apache.org/jira/browse/FLINK-16696

Regards,
Roman

On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <[hidden email]> wrote:

Hey Flink Community,

I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be available via the REST API. Does this exist today?

Aaron

[Ufuk Celebi]

Hey Aaron,

you can expect one of the two responses for COMPLETED savepoints [1, 2].

1. Success

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "location": "string"
  }
}

2. Failure

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "failure-cause": {
      "class": "string",
      "stack-trace": "string",
      "serialized-throwable": "string"
    }
  }
}

– Ufuk

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
[2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java

On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <[hidden email]> wrote:

>
> Roman,
>
> Thanks for the info. That's super helpful. I'd be interested in picking that ticket up.
>
> One additional question: the states that can return from this API are only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures represented for this endpoint?
>
> Aaron
>
> On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <[hidden email]> wrote:
>>
>> Hey Aaron,
>>
>> You can use /jobs/:jobid/savepoints/:triggerid to get the location when the checkpoint is completed.
>>
>> Please see https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>>
>> Meanwhile, I've created an issue to update the docs: https://issues.apache.org/jira/browse/FLINK-16696
>>
>> Regards,
>> Roman
>>
>>
>> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <[hidden email]> wrote:
>>>
>>> Hey Flink Community,
>>>
>>> I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be available via the REST API. Does this exist today?
>>>
>>> Aaron

[Aaron Langford]

All, it looks like the actual return structure from the API is:

1. Success

{
  "status": {
    "id": "completed"
  },
  "operation": {
    "location": "string"
  }
}

2. Failure

{
  "status": {
    "id": "completed"
  },
  "operation": {
    "failure-cause": {
      "class": "string",
      "stack-trace": "string",
      "serialized-throwable": "string"
    }
  }
}

The difference is that "operation" is the key in the response, not "savepoint". Should this go into the FLINK-16696 ticket or a separate one?

Aaron

On Fri, Mar 20, 2020 at 1:33 PM Ufuk Celebi <[hidden email]> wrote:

Hey Aaron,

you can expect one of the two responses for COMPLETED savepoints [1, 2].

1. Success

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "location": "string"
  }
}

2. Failure

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "failure-cause": {
      "class": "string",
      "stack-trace": "string",
      "serialized-throwable": "string"
    }
  }
}

– Ufuk

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
[2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java

On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <[hidden email]> wrote:

>
> Roman,
>
> Thanks for the info. That's super helpful. I'd be interested in picking that ticket up.
>
> One additional question: the states that can return from this API are only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures represented for this endpoint?
>
> Aaron
>
> On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <[hidden email]> wrote:
>>
>> Hey Aaron,
>>
>> You can use /jobs/:jobid/savepoints/:triggerid to get the location when the checkpoint is completed.
>>
>> Please see https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>>
>> Meanwhile, I've created an issue to update the docs: https://issues.apache.org/jira/browse/FLINK-16696
>>
>> Regards,
>> Roman
>>
>>
>> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <[hidden email]> wrote:
>>>
>>> Hey Flink Community,
>>>
>>> I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be available via the REST API. Does this exist today?
>>>
>>> Aaron

[rmetzger0]

Good catch!. Yes, you can add this to FLINK-16696.

On Wed, Apr 1, 2020 at 10:59 PM Aaron Langford <[hidden email]> wrote:

All, it looks like the actual return structure from the API is:

1. Success

{
  "status": {
    "id": "completed"
  },
  "operation": {
    "location": "string"
  }
}

2. Failure

{
  "status": {
    "id": "completed"
  },
  "operation": {
    "failure-cause": {
      "class": "string",
      "stack-trace": "string",
      "serialized-throwable": "string"
    }
  }
}

The difference is that "operation" is the key in the response, not "savepoint". Should this go into the FLINK-16696 ticket or a separate one?

Aaron

On Fri, Mar 20, 2020 at 1:33 PM Ufuk Celebi <[hidden email]> wrote:

Hey Aaron,

you can expect one of the two responses for COMPLETED savepoints [1, 2].

1. Success

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "location": "string"
  }
}

2. Failure

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "failure-cause": {
      "class": "string",
      "stack-trace": "string",
      "serialized-throwable": "string"
    }
  }
}

– Ufuk

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
[2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java

On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <[hidden email]> wrote:

>
> Roman,
>
> Thanks for the info. That's super helpful. I'd be interested in picking that ticket up.
>
> One additional question: the states that can return from this API are only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures represented for this endpoint?
>
> Aaron
>
> On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <[hidden email]> wrote:
>>
>> Hey Aaron,
>>
>> You can use /jobs/:jobid/savepoints/:triggerid to get the location when the checkpoint is completed.
>>
>> Please see https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>>
>> Meanwhile, I've created an issue to update the docs: https://issues.apache.org/jira/browse/FLINK-16696
>>
>> Regards,
>> Roman
>>
>>
>> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <[hidden email]> wrote:
>>>
>>> Hey Flink Community,
>>>
>>> I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be available via the REST API. Does this exist today?
>>>
>>> Aaron

[Ufuk Celebi]

Sorry for the copy & paste error in my earlier message. 🙄 I agree with Robert.

On 2. Apr 2020, at 11:06, Robert Metzger <[hidden email]> wrote:

Good catch!. Yes, you can add this to FLINK-16696.

On Wed, Apr 1, 2020 at 10:59 PM Aaron Langford <[hidden email]> wrote:

All, it looks like the actual return structure from the API is:

1. Success

{
  "status": {
    "id": "completed"
  },
  "operation": {
    "location": "string"
  }
}

2. Failure

{
  "status": {
    "id": "completed"
  },
  "operation": {
    "failure-cause": {
      "class": "string",
      "stack-trace": "string",
      "serialized-throwable": "string"
    }
  }
}

The difference is that "operation" is the key in the response, not "savepoint". Should this go into the FLINK-16696 ticket or a separate one?

Aaron

On Fri, Mar 20, 2020 at 1:33 PM Ufuk Celebi <[hidden email]> wrote:

Hey Aaron,

you can expect one of the two responses for COMPLETED savepoints [1, 2].

1. Success

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "location": "string"
  }
}

2. Failure

{
  "status": {
    "id": "completed"
  },
  "savepoint": {
    "failure-cause": {
      "class": "string",
      "stack-trace": "string",
      "serialized-throwable": "string"
    }
  }
}

– Ufuk

[1] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.java#L209-L217
[2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/json/SerializedThrowableSerializer.java

On Fri, Mar 20, 2020 at 7:22 PM Aaron Langford <[hidden email]> wrote:

>
> Roman,
>
> Thanks for the info. That's super helpful. I'd be interested in picking that ticket up.
>
> One additional question: the states that can return from this API are only described as 'COMPLETED' or 'IN_PROGRESS'. How are failures represented for this endpoint?
>
> Aaron
>
> On Fri, Mar 20, 2020 at 2:29 AM Khachatryan Roman <[hidden email]> wrote:
>>
>> Hey Aaron,
>>
>> You can use /jobs/:jobid/savepoints/:triggerid to get the location when the checkpoint is completed.
>>
>> Please see https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/index.html?org/apache/flink/runtime/rest/handler/job/savepoints/SavepointHandlers.html
>>
>> Meanwhile, I've created an issue to update the docs: https://issues.apache.org/jira/browse/FLINK-16696
>>
>> Regards,
>> Roman
>>
>>
>> On Fri, Mar 20, 2020 at 5:09 AM Aaron Langford <[hidden email]> wrote:
>>>
>>> Hey Flink Community,
>>>
>>> I'm combing through docs right now, and I don't see that a savepoint location is returned or surfaced anywhere. When I do this in the CLI, I get a nice message that tells me where in S3 it put my savepoint (unique savepoint ID included). I'm looking for that same result to be available via the REST API. Does this exist today?
>>>
>>> Aaron