[External] checkpoint metadata not in configured directory state.checkpoints.dir

[Vishal Sharma]

Hi Folks, 

I am using flink 1.8 with externalised checkpointing enabled and saving the checkpoints to aws S3. 

My configuration is as follows :

flink-conf.yaml :

state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata

In application code :
env.setStateBackend(new RocksDBStateBackend("s3a://test-bucket/checkpoints", true))

As per my understanding, the externalized checkpoint’s meta data is determined from the configuration key "state.checkpoints.dir" and checkpoint data is stored in state backend path. 




However, In my case, I don't see anything in the metadata directory. The _metadata file is present inside each of the checkpoint directory (chk-6043 ...). 





Is this the expected behavior ? If yes, what is the use of "state.checkpoints.dir" configuration ?

My goal is to establish a process to automatically restart the job from last completed externalised checkpoint in case of failure. For this to happen, I need to able to figure out path for the metadata of latest checkpoint.

Thanks, 

Vishal Sharma

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Chesnay Schepler]

The _metadata is always stored in the same directory as the checkpoint data.

As outlined here "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ be overwritten with a job-specific setting when creating the state-backend.

If you want the state-backend to use the configured directory you must configure the state-backend in the configuration as well, as outlined here.

On 19/06/2019 16:26, Vishal Sharma wrote:

Hi Folks, 

I am using flink 1.8 with externalised checkpointing enabled and saving the checkpoints to aws S3. 

My configuration is as follows :

flink-conf.yaml :

state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata

In application code :
env.setStateBackend(new RocksDBStateBackend("s3a://test-bucket/checkpoints", true))

As per my understanding, the externalized checkpoint’s meta data is determined from the configuration key "state.checkpoints.dir" and checkpoint data is stored in state backend path. 




However, In my case, I don't see anything in the metadata directory. The _metadata file is present inside each of the checkpoint directory (chk-6043 ...). 





Is this the expected behavior ? If yes, what is the use of "state.checkpoints.dir" configuration ?

My goal is to establish a process to automatically restart the job from last completed externalised checkpoint in case of failure. For this to happen, I need to able to figure out path for the metadata of latest checkpoint.

Thanks, 

Vishal Sharma

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Vishal Sharma]

Hi Chesnay, 

Can you suggest, How should I go about automating job restart from last completed externalised checkpoint in case of failure ? I am not sure about the path for the latest completed checkpoint. 

Thanks, 

Vishal Sharma

On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <[hidden email]> wrote:

The _metadata is always stored in the same directory as the checkpoint data.

As outlined here "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ be overwritten with a job-specific setting when creating the state-backend.

If you want the state-backend to use the configured directory you must configure the state-backend in the configuration as well, as outlined here.

On 19/06/2019 16:26, Vishal Sharma wrote:

Hi Folks, 

I am using flink 1.8 with externalised checkpointing enabled and saving the checkpoints to aws S3. 

My configuration is as follows :

flink-conf.yaml :

state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata

In application code :
env.setStateBackend(new RocksDBStateBackend("s3a://test-bucket/checkpoints", true))

As per my understanding, the externalized checkpoint’s meta data is determined from the configuration key "state.checkpoints.dir" and checkpoint data is stored in state backend path. 




However, In my case, I don't see anything in the metadata directory. The _metadata file is present inside each of the checkpoint directory (chk-6043 ...). 





Is this the expected behavior ? If yes, what is the use of "state.checkpoints.dir" configuration ?

My goal is to establish a process to automatically restart the job from last completed externalised checkpoint in case of failure. For this to happen, I need to able to figure out path for the metadata of latest checkpoint.

Thanks, 

Vishal Sharma

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Congxian Qiu]

Hi, Vishal

If you want to restart from the last competed external checkpoint of the previous stoped job, you need to track the checkpoint path and restart from it.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint

Vishal Sharma <[hidden email]> 于2019年6月19日周三 下午11:38写道：

Hi Chesnay, 

Can you suggest, How should I go about automating job restart from last completed externalised checkpoint in case of failure ? I am not sure about the path for the latest completed checkpoint. 

Thanks, 

Vishal Sharma

On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <[hidden email]> wrote:

The _metadata is always stored in the same directory as the checkpoint data.

As outlined here "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ be overwritten with a job-specific setting when creating the state-backend.

If you want the state-backend to use the configured directory you must configure the state-backend in the configuration as well, as outlined here.

On 19/06/2019 16:26, Vishal Sharma wrote:

Hi Folks, 

I am using flink 1.8 with externalised checkpointing enabled and saving the checkpoints to aws S3. 

My configuration is as follows :

flink-conf.yaml :

state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata

In application code :
env.setStateBackend(new RocksDBStateBackend("s3a://test-bucket/checkpoints", true))

As per my understanding, the externalized checkpoint’s meta data is determined from the configuration key "state.checkpoints.dir" and checkpoint data is stored in state backend path. 




However, In my case, I don't see anything in the metadata directory. The _metadata file is present inside each of the checkpoint directory (chk-6043 ...). 





Is this the expected behavior ? If yes, what is the use of "state.checkpoints.dir" configuration ?

My goal is to establish a process to automatically restart the job from last completed externalised checkpoint in case of failure. For this to happen, I need to able to figure out path for the metadata of latest checkpoint.

Thanks, 

Vishal Sharma

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Vishal Sharma]

Hi Congxian, 

I am not sure how can I track the checkpoint path. Can you suggestion regarding this ?

Thanks, 

Vishal Sharma

On Thu, Jun 20, 2019 at 11:17 AM Congxian Qiu <[hidden email]> wrote:

Hi, Vishal

If you want to restart from the last competed external checkpoint of the previous stoped job, you need to track the checkpoint path and restart from it.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint

Vishal Sharma <[hidden email]> 于2019年6月19日周三 下午11:38写道：

Hi Chesnay, 

Can you suggest, How should I go about automating job restart from last completed externalised checkpoint in case of failure ? I am not sure about the path for the latest completed checkpoint. 

Thanks, 

Vishal Sharma

On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <[hidden email]> wrote:

The _metadata is always stored in the same directory as the checkpoint data.

As outlined here "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ be overwritten with a job-specific setting when creating the state-backend.

If you want the state-backend to use the configured directory you must configure the state-backend in the configuration as well, as outlined here.

On 19/06/2019 16:26, Vishal Sharma wrote:

Hi Folks, 

I am using flink 1.8 with externalised checkpointing enabled and saving the checkpoints to aws S3. 

My configuration is as follows :

flink-conf.yaml :

state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata

In application code :
env.setStateBackend(new RocksDBStateBackend("s3a://test-bucket/checkpoints", true))

As per my understanding, the externalized checkpoint’s meta data is determined from the configuration key "state.checkpoints.dir" and checkpoint data is stored in state backend path. 




However, In my case, I don't see anything in the metadata directory. The _metadata file is present inside each of the checkpoint directory (chk-6043 ...). 





Is this the expected behavior ? If yes, what is the use of "state.checkpoints.dir" configuration ?

My goal is to establish a process to automatically restart the job from last completed externalised checkpoint in case of failure. For this to happen, I need to able to figure out path for the metadata of latest checkpoint.

Thanks, 

Vishal Sharma

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Congxian Qiu]

Hi Vishal 

Maybe the slide[1] (page 40) can be helpful

[1] https://files.alicdn.com/tpsservice/c421720fcb1c51026257cd770923844a.pdf

Vishal Sharma <[hidden email]> 于2019年6月20日周四 下午5:42写道：

Hi Congxian, 

I am not sure how can I track the checkpoint path. Can you suggestion regarding this ?

Thanks, 

Vishal Sharma

On Thu, Jun 20, 2019 at 11:17 AM Congxian Qiu <[hidden email]> wrote:

Hi, Vishal

If you want to restart from the last competed external checkpoint of the previous stoped job, you need to track the checkpoint path and restart from it.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint

Vishal Sharma <[hidden email]> 于2019年6月19日周三 下午11:38写道：

Hi Chesnay, 

Can you suggest, How should I go about automating job restart from last completed externalised checkpoint in case of failure ? I am not sure about the path for the latest completed checkpoint. 

Thanks, 

Vishal Sharma

On Wed, Jun 19, 2019 at 11:11 PM Chesnay Schepler <[hidden email]> wrote:

The _metadata is always stored in the same directory as the checkpoint data.

As outlined here "state.checkpoints.dir" serves as a cluster-wide configuration that _can_ be overwritten with a job-specific setting when creating the state-backend.

If you want the state-backend to use the configured directory you must configure the state-backend in the configuration as well, as outlined here.

On 19/06/2019 16:26, Vishal Sharma wrote:

Hi Folks, 

I am using flink 1.8 with externalised checkpointing enabled and saving the checkpoints to aws S3. 

My configuration is as follows :

flink-conf.yaml :

state.checkpoints.dir: s3a://test-bucket/checkpoint-metadata

In application code :
env.setStateBackend(new RocksDBStateBackend("s3a://test-bucket/checkpoints", true))

As per my understanding, the externalized checkpoint’s meta data is determined from the configuration key "state.checkpoints.dir" and checkpoint data is stored in state backend path. 




However, In my case, I don't see anything in the metadata directory. The _metadata file is present inside each of the checkpoint directory (chk-6043 ...). 





Is this the expected behavior ? If yes, what is the use of "state.checkpoints.dir" configuration ?

My goal is to establish a process to automatically restart the job from last completed externalised checkpoint in case of failure. For this to happen, I need to able to figure out path for the metadata of latest checkpoint.

Thanks, 

Vishal Sharma

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

Grab is hiring. Learn more at https://grab.careers

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.