Query on job restoration using relocated savepoint

[Parth Sarathy]

Hi All,
               We are trying to restore a job using relocated savepoint
files. As pointed out in the FAQs of savepoint documentation, savepoints
have absolute paths recorded in them and hence a simple relocation to
restore the job would fail. As directed in the documentation we tried out
the simple way to refactor the paths by editing them manually, but the job
submission failed with an IllegalStateException as noted below :
Caused by: java.lang.IllegalStateException: Reading invalid
OperatorStateHandle, type: 50
        at
org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.deserializeOperatorStateHandle(SavepointV2Serializer.java:499)

We then went ahead and gave a swing at the second prescribed option of
utilizing the SavepointV2Serializer for deserializing and serializing the
metadata file. Even with this approach we observed that the generated
metadata file still referenced the old absolute path. We are in a stuck in a
predicament as of now. How is it that we can set / change the absolute paths
present in the metadata file using the prescribed SavepointV2Serializer.
It’d be helpful if you could provide some insight into this.

Thanks,
Parth Sarathy



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Till Rohrmann-2]

Hi Parth,

I've pulled Stefan into the conversation who might be able to help you with your problem.

Cheers,

Till

On Wed, Apr 10, 2019 at 7:17 PM Parth Sarathy <[hidden email]> wrote:

Hi All,
               We are trying to restore a job using relocated savepoint
files. As pointed out in the FAQs of savepoint documentation, savepoints
have absolute paths recorded in them and hence a simple relocation to
restore the job would fail. As directed in the documentation we tried out
the simple way to refactor the paths by editing them manually, but the job
submission failed with an IllegalStateException as noted below :
Caused by: java.lang.IllegalStateException: Reading invalid
OperatorStateHandle, type: 50
        at
org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.deserializeOperatorStateHandle(SavepointV2Serializer.java:499)

We then went ahead and gave a swing at the second prescribed option of
utilizing the SavepointV2Serializer for deserializing and serializing the
metadata file. Even with this approach we observed that the generated
metadata file still referenced the old absolute path. We are in a stuck in a
predicament as of now. How is it that we can set / change the absolute paths
present in the metadata file using the prescribed SavepointV2Serializer.
It’d be helpful if you could provide some insight into this.

Thanks,
Parth Sarathy



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Stefan Richter-4]

Hi,

the first case sounds like you made a mistake when editing the paths manually and deleted one ore more bytes that were not part of the path and thus corrupted the meta data. For the second approach, of course you also need to replace the paths after reading and before rewriting the metadata. This approach is basically the programmatic version of your first attempt, but using Flink’s code to avoid the pitfalls of corrupting the file.

Best,
Stefan

> On 10. Apr 2019, at 19:16, Parth Sarathy <[hidden email]> wrote:
>
> Hi All,
>               We are trying to restore a job using relocated savepoint
> files. As pointed out in the FAQs of savepoint documentation, savepoints
> have absolute paths recorded in them and hence a simple relocation to
> restore the job would fail. As directed in the documentation we tried out
> the simple way to refactor the paths by editing them manually, but the job
> submission failed with an IllegalStateException as noted below :
> Caused by: java.lang.IllegalStateException: Reading invalid
> OperatorStateHandle, type: 50
>        at
> org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.deserializeOperatorStateHandle(SavepointV2Serializer.java:499)
>
> We then went ahead and gave a swing at the second prescribed option of
> utilizing the SavepointV2Serializer for deserializing and serializing the
> metadata file. Even with this approach we observed that the generated
> metadata file still referenced the old absolute path. We are in a stuck in a
> predicament as of now. How is it that we can set / change the absolute paths
> present in the metadata file using the prescribed SavepointV2Serializer.
> It’d be helpful if you could provide some insight into this.
>
> Thanks,
> Parth Sarathy
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Stefan Richter-4]

Small correction, on the first case: more likely is that you changed the path string but I think those are prefixed by the string length, so that would require manual adjustment as well to not corrupt the metadata.

> On 11. Apr 2019, at 14:42, Stefan Richter <[hidden email]> wrote:
>
> Hi,
>
> the first case sounds like you made a mistake when editing the paths manually and deleted one ore more bytes that were not part of the path and thus corrupted the meta data. For the second approach, of course you also need to replace the paths after reading and before rewriting the metadata. This approach is basically the programmatic version of your first attempt, but using Flink’s code to avoid the pitfalls of corrupting the file.
>
> Best,
> Stefan
>
>> On 10. Apr 2019, at 19:16, Parth Sarathy <[hidden email]> wrote:
>>
>> Hi All,
>>              We are trying to restore a job using relocated savepoint
>> files. As pointed out in the FAQs of savepoint documentation, savepoints
>> have absolute paths recorded in them and hence a simple relocation to
>> restore the job would fail. As directed in the documentation we tried out
>> the simple way to refactor the paths by editing them manually, but the job
>> submission failed with an IllegalStateException as noted below :
>> Caused by: java.lang.IllegalStateException: Reading invalid
>> OperatorStateHandle, type: 50
>>       at
>> org.apache.flink.runtime.checkpoint.savepoint.SavepointV2Serializer.deserializeOperatorStateHandle(SavepointV2Serializer.java:499)
>>
>> We then went ahead and gave a swing at the second prescribed option of
>> utilizing the SavepointV2Serializer for deserializing and serializing the
>> metadata file. Even with this approach we observed that the generated
>> metadata file still referenced the old absolute path. We are in a stuck in a
>> predicament as of now. How is it that we can set / change the absolute paths
>> present in the metadata file using the prescribed SavepointV2Serializer.
>> It’d be helpful if you could provide some insight into this.
>>
>> Thanks,
>> Parth Sarathy
>>
>>
>>
>> --
>> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

[Parth Sarathy]

Hi Stefan,
               Thank you for analyzing the issue and providing a quick
reply. We are looking to go ahead with the programmatic method of rewriting
the metadata with the new path. We were successfully able to de-serialize
and serialize the metadata to a new path, but still we see that the path
inside the file fails to change after serializing back. We are not seeing
any definitive way to modify the path after de-serializing the metadata. Can
you please guide us on how we can alter the path after de-serializing the
metadata file.

Thanks,
Parth sarathy




--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/