(DEPRECATED) Apache Flink User Mailing List archive.

Savepoint documentation

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
Farouk
Reply | Threaded
Open this post in threaded view
♦
♦
|

Savepoint documentation

Farouk
Hi


Can I move the Savepoint files on stable storage?

The quick answer to this question is currently “no” because the meta data file references the files on stable storage as absolute paths for technical reasons. The longer answer is: if you MUST move the files for some reason there are two potential approaches as workaround. First, simpler but potentially more dangerous, you can use an editor to find the old path in the meta data file and replace them with the new path. Second, you can use the class SavepointV2Serializer as starting point to programmatically read, manipulate, and rewrite the meta data file with the new paths.


Thanks
Farouk
David Anderson-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Savepoint documentation

David Anderson-4
You are correct in thinking that the documentation wasn't updated. If you look at the master docs [1] you will see that they now say

Can I move the Savepoint files on stable storage? #

The quick answer to this question is currently “yes”. Sink Flink 1.11.0, savepoints are self-contained and relocatable. You can move the file and restore from any location.


On Tue, Mar 2, 2021 at 10:33 AM Farouk <[hidden email]> wrote:
Hi


Can I move the Savepoint files on stable storage?

The quick answer to this question is currently “no” because the meta data file references the files on stable storage as absolute paths for technical reasons. The longer answer is: if you MUST move the files for some reason there are two potential approaches as workaround. First, simpler but potentially more dangerous, you can use an editor to find the old path in the meta data file and replace them with the new path. Second, you can use the class SavepointV2Serializer as starting point to programmatically read, manipulate, and rewrite the meta data file with the new paths.


Thanks
Farouk
XU Qinghui-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Savepoint documentation

XU Qinghui-2
Out of curiosity, does it mean that savepoint created by flink 1.11 cannot be recovered by a job running with flink 1.10 or older versions (so downgrade is impossible)?

Le mar. 2 mars 2021 à 12:25, David Anderson <[hidden email]> a écrit :
You are correct in thinking that the documentation wasn't updated. If you look at the master docs [1] you will see that they now say

Can I move the Savepoint files on stable storage? #

The quick answer to this question is currently “yes”. Sink Flink 1.11.0, savepoints are self-contained and relocatable. You can move the file and restore from any location.


On Tue, Mar 2, 2021 at 10:33 AM Farouk <[hidden email]> wrote:
Hi


Can I move the Savepoint files on stable storage?

The quick answer to this question is currently “no” because the meta data file references the files on stable storage as absolute paths for technical reasons. The longer answer is: if you MUST move the files for some reason there are two potential approaches as workaround. First, simpler but potentially more dangerous, you can use an editor to find the old path in the meta data file and replace them with the new path. Second, you can use the class SavepointV2Serializer as starting point to programmatically read, manipulate, and rewrite the meta data file with the new paths.


Thanks
Farouk
David Anderson-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Savepoint documentation

David Anderson-4
> Out of curiosity, does it mean that savepoint created by flink 1.11 cannot be recovered by a job running with flink 1.10 or older versions (so downgrade is impossible)?

That's correct. See the mailing list thread on Backwards Compatibility of Savepoints [1].


On Tue, Mar 2, 2021 at 4:39 PM XU Qinghui <[hidden email]> wrote:
Out of curiosity, does it mean that savepoint created by flink 1.11 cannot be recovered by a job running with flink 1.10 or older versions (so downgrade is impossible)?

Le mar. 2 mars 2021 à 12:25, David Anderson <[hidden email]> a écrit :
You are correct in thinking that the documentation wasn't updated. If you look at the master docs [1] you will see that they now say

Can I move the Savepoint files on stable storage? #

The quick answer to this question is currently “yes”. Sink Flink 1.11.0, savepoints are self-contained and relocatable. You can move the file and restore from any location.


On Tue, Mar 2, 2021 at 10:33 AM Farouk <[hidden email]> wrote:
Hi


Can I move the Savepoint files on stable storage?

The quick answer to this question is currently “no” because the meta data file references the files on stable storage as absolute paths for technical reasons. The longer answer is: if you MUST move the files for some reason there are two potential approaches as workaround. First, simpler but potentially more dangerous, you can use an editor to find the old path in the meta data file and replace them with the new path. Second, you can use the class SavepointV2Serializer as starting point to programmatically read, manipulate, and rewrite the meta data file with the new paths.


Thanks
Farouk
Farouk
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Savepoint documentation

Farouk
Thanks a million :)

Le mer. 3 mars 2021 à 11:15, David Anderson <[hidden email]> a écrit :
> Out of curiosity, does it mean that savepoint created by flink 1.11 cannot be recovered by a job running with flink 1.10 or older versions (so downgrade is impossible)?

That's correct. See the mailing list thread on Backwards Compatibility of Savepoints [1].


On Tue, Mar 2, 2021 at 4:39 PM XU Qinghui <[hidden email]> wrote:
Out of curiosity, does it mean that savepoint created by flink 1.11 cannot be recovered by a job running with flink 1.10 or older versions (so downgrade is impossible)?

Le mar. 2 mars 2021 à 12:25, David Anderson <[hidden email]> a écrit :
You are correct in thinking that the documentation wasn't updated. If you look at the master docs [1] you will see that they now say

Can I move the Savepoint files on stable storage? #

The quick answer to this question is currently “yes”. Sink Flink 1.11.0, savepoints are self-contained and relocatable. You can move the file and restore from any location.


On Tue, Mar 2, 2021 at 10:33 AM Farouk <[hidden email]> wrote:
Hi


Can I move the Savepoint files on stable storage?

The quick answer to this question is currently “no” because the meta data file references the files on stable storage as absolute paths for technical reasons. The longer answer is: if you MUST move the files for some reason there are two potential approaches as workaround. First, simpler but potentially more dangerous, you can use an editor to find the old path in the meta data file and replace them with the new path. Second, you can use the class SavepointV2Serializer as starting point to programmatically read, manipulate, and rewrite the meta data file with the new paths.


Thanks
Farouk
Free forum by Nabble Edit this page