Flink 1.10 -> Savepoints referencing to checkpoints or not

[Bajaj, Abhinav]

Hi,

 

I am trying to understand the Flink 1.10 savepoints related documentation that mentions -

“When triggering a savepoint, a new savepoint directory is created where the data as well as the meta data will be stored.”

 

However, I came across this issue - https://issues.apache.org/jira/browse/FLINK-5763 which mentions the problem -

“For file system based checkpoints (FsStateBackend, RocksDBStateBackend) this results in the savepoint referencing files from the checkpoint directory (usually different than <target>)”

 

My understanding of the issue is that savepoints were not storing the data and were pointing to checkpoints for actual state data.

The issue has been fixed and changes made into Flink 1.11.

 

But, I would like to get the behavior for Flink 1.10 confirmed.

 

Does savepoint in Flink 1.10 store both the data & metadata? OR it has references to checkpoints & need access to checkpoints directory to restore state?

 

Thanks much,

Abhinav Bajaj

[Congxian Qiu]

Hi  Bajaj

     Savepoint does contain the metadata and data in Flink 1.10, it does not need to reference any checkpoint data.

Bajaj, Abhinav <[hidden email]> 于2020年11月17日周二 上午8:58写道：

Hi,

 

I am trying to understand the Flink 1.10 savepoints related documentation that mentions -

“When triggering a savepoint, a new savepoint directory is created where the data as well as the meta data will be stored.”

 

However, I came across this issue - https://issues.apache.org/jira/browse/FLINK-5763 which mentions the problem -

“For file system based checkpoints (FsStateBackend, RocksDBStateBackend) this results in the savepoint referencing files from the checkpoint directory (usually different than <target>)”

 

My understanding of the issue is that savepoints were not storing the data and were pointing to checkpoints for actual state data.

The issue has been fixed and changes made into Flink 1.11.

 

But, I would like to get the behavior for Flink 1.10 confirmed.

 

Does savepoint in Flink 1.10 store both the data & metadata? OR it has references to checkpoints & need access to checkpoints directory to restore state?

 

Thanks much,

Abhinav Bajaj

[Tzu-Li (Gordon) Tai]

Hi,

Both the data and metadata is being stored in the savepoint directory, since
Flink 1.3.
The metadata in the savepoint directory does not reference and checkpoint
data files.

In 1.11, what was changed was that the savepoint metadata uses relative
paths to point to the data files in the savepoint directory, instead of
absolute paths.
This would allow you to treat the savepoint directories as self-contained
and free to relocate with filesystem operations outside of Flink.

Hope this clarifies things for you!

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/