(DEPRECATED) Apache Flink User Mailing List archive.

Saved State in FSstate Backend

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Abdullah bin Omar
Reply | Threaded
Open this post in threaded view
♦
♦
|

Saved State in FSstate Backend

Abdullah bin Omar
Hi,

I used the FSstate backend to save the state. I just got a folder named similar to JobID (attached image). Inside the folder, there are two more folders named by shared and task owned. However, there is nothing in those folders.

How can I see the saved state? or, where is the state saved?

Thank you! 

Screen Shot 2021-03-17 at 4.15.39 AM.png (55K) Download Attachment
Screen Shot 2021-03-17 at 4.15.31 AM.png (39K) Download Attachment
Yun Tang
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Saved State in FSstate Backend

Yun Tang
Hi

You could refer to [1] to know more details about checkpoint directory structure. If you are using FsStateBackend, all checkpointed data should be found under 'exclusive' folder, and nothing would exist if keyed state handle smaller than memory threshold [2] (checkpointed data would be sent to JM directly).



Best
Yun Tang
From: Abdullah bin Omar <[hidden email]>
Sent: Wednesday, March 17, 2021 17:41
To: [hidden email] <[hidden email]>
Subject: Saved State in FSstate Backend
 
Hi,

I used the FSstate backend to save the state. I just got a folder named similar to JobID (attached image). Inside the folder, there are two more folders named by shared and task owned. However, there is nothing in those folders.

How can I see the saved state? or, where is the state saved?

Thank you! 
Abdullah bin Omar
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Saved State in FSstate Backend

Abdullah bin Omar
Hi, 

I use the checkpoint configuration 10000 ms. 

env.getCheckpointConfig().setCheckpointTimeout(10000);


used FSstatebackend to save the state

   env.setStateBackend(new FsStateBackend("file:///Users/username/Documents/checkpoint-save-java/checkpointing.txt", false)); //here I tried both true and false 


After running the code,the job is finished. It gives a path in the checkpoint detail section of the apache flink UI:


checkpoint detail:

path: file:/Users/username/Documents/checkpoint-save-java/checkpointing.txt/7e6236fea7b572e6f06595ba11ec0eed/chk-58

discarded: true

checkpoint type: aligned checkpoint


In Summary (of the checkpoint section of apache flink UI):


End to End duration (for maximum): 133 ms

Checkpointed data size: 2.82 KB



Everything is okay. The code runs well. The job is finished, and the checkpoint interval is more than to complete each checkpoint. 



Question:

Why is discarded shown true? and why still there is no file appeared in the location of the FsStateBackend?



Thank you!


Regards,

Abdullah


On Fri, Mar 19, 2021 at 8:26 AM Abdullah bin Omar <[hidden email]> wrote:
Hi Yun, 

I use the checkpoint configuration 10000 ms. 

env.getCheckpointConfig().setCheckpointTimeout(10000);


used FSstatebackend to save the state

   env.setStateBackend(new FsStateBackend("file:///Users/username/Documents/checkpoint-save-java/checkpointing.txt", false)); //here I tried both true and false 


After running the code,the job is finished. It gives a path in the checkpoint detail section of the apache flink UI:


checkpoint detail:

path: file:/Users/username/Documents/checkpoint-save-java/checkpointing.txt/7e6236fea7b572e6f06595ba11ec0eed/chk-58

discarded: true

checkpoint type: aligned checkpoint


In Summary (of the checkpoint section of apache flink UI):


End to End duration (for maximum): 133 ms

Checkpointed data size: 2.82 KB



Everything is okay. The code runs well. The job is finished, and the checkpoint interval is more than to complete each checkpoint. 



Question:

Why is discarded shown true? and why still there is no file appeared in the location of the FsStateBackend?



Thank you!


Regards,

Abdullah







On Wed, Mar 17, 2021 at 9:16 AM Yun Tang <[hidden email]> wrote:
Hi

You could refer to [1] to know more details about checkpoint directory structure. If you are using FsStateBackend, all checkpointed data should be found under 'exclusive' folder, and nothing would exist if keyed state handle smaller than memory threshold [2] (checkpointed data would be sent to JM directly).



Best
Yun Tang
From: Abdullah bin Omar <[hidden email]>
Sent: Wednesday, March 17, 2021 17:41
To: [hidden email] <[hidden email]>
Subject: Saved State in FSstate Backend
 
Hi,

I used the FSstate backend to save the state. I just got a folder named similar to JobID (attached image). Inside the folder, there are two more folders named by shared and task owned. However, there is nothing in those folders.

How can I see the saved state? or, where is the state saved?

Thank you! 
Arvid Heise-4
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Saved State in FSstate Backend

Arvid Heise-4
All checkpoints are managed by the Checkpoint Coordinator and deleted once the job stops. You can use retained checkpoints [1] to keep them after job termination. You can also use stop-with-savepoint to create a savepoint [2].


On Fri, Mar 19, 2021 at 2:53 PM Abdullah bin Omar <[hidden email]> wrote:
Hi, 

I use the checkpoint configuration 10000 ms. 

env.getCheckpointConfig().setCheckpointTimeout(10000);


used FSstatebackend to save the state

   env.setStateBackend(new FsStateBackend("file:///Users/username/Documents/checkpoint-save-java/checkpointing.txt", false)); //here I tried both true and false 


After running the code,the job is finished. It gives a path in the checkpoint detail section of the apache flink UI:


checkpoint detail:

path: file:/Users/username/Documents/checkpoint-save-java/checkpointing.txt/7e6236fea7b572e6f06595ba11ec0eed/chk-58

discarded: true

checkpoint type: aligned checkpoint


In Summary (of the checkpoint section of apache flink UI):


End to End duration (for maximum): 133 ms

Checkpointed data size: 2.82 KB



Everything is okay. The code runs well. The job is finished, and the checkpoint interval is more than to complete each checkpoint. 



Question:

Why is discarded shown true? and why still there is no file appeared in the location of the FsStateBackend?



Thank you!


Regards,

Abdullah


On Fri, Mar 19, 2021 at 8:26 AM Abdullah bin Omar <[hidden email]> wrote:
Hi Yun, 

I use the checkpoint configuration 10000 ms. 

env.getCheckpointConfig().setCheckpointTimeout(10000);


used FSstatebackend to save the state

   env.setStateBackend(new FsStateBackend("file:///Users/username/Documents/checkpoint-save-java/checkpointing.txt", false)); //here I tried both true and false 


After running the code,the job is finished. It gives a path in the checkpoint detail section of the apache flink UI:


checkpoint detail:

path: file:/Users/username/Documents/checkpoint-save-java/checkpointing.txt/7e6236fea7b572e6f06595ba11ec0eed/chk-58

discarded: true

checkpoint type: aligned checkpoint


In Summary (of the checkpoint section of apache flink UI):


End to End duration (for maximum): 133 ms

Checkpointed data size: 2.82 KB



Everything is okay. The code runs well. The job is finished, and the checkpoint interval is more than to complete each checkpoint. 



Question:

Why is discarded shown true? and why still there is no file appeared in the location of the FsStateBackend?



Thank you!


Regards,

Abdullah







On Wed, Mar 17, 2021 at 9:16 AM Yun Tang <[hidden email]> wrote:
Hi

You could refer to [1] to know more details about checkpoint directory structure. If you are using FsStateBackend, all checkpointed data should be found under 'exclusive' folder, and nothing would exist if keyed state handle smaller than memory threshold [2] (checkpointed data would be sent to JM directly).



Best
Yun Tang
From: Abdullah bin Omar <[hidden email]>
Sent: Wednesday, March 17, 2021 17:41
To: [hidden email] <[hidden email]>
Subject: Saved State in FSstate Backend
 
Hi,

I used the FSstate backend to save the state. I just got a folder named similar to JobID (attached image). Inside the folder, there are two more folders named by shared and task owned. However, there is nothing in those folders.

How can I see the saved state? or, where is the state saved?

Thank you! 
Free forum by Nabble Edit this page