Apache Flink - Operator name and uuid best practices

[M Singh]

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans

[bupt_ljy]

Hi Mans!

Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1].

> Are there any restrictions on the length of the name and uuid attributes?

It’s pretty much the same as you define a string value, so there is no special restrictions on this.

> Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

I’m not a hundred percent sure about this but I run a testing program and it works fine.

> Can the name and uuid be the same ? 

Yes. But uuids accross operators cannot be same.

For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling.

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best,

Jiayi Liao

At 2019-11-16 18:35:38, "M Singh" <[hidden email]> wrote:

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans

 

[M Singh]

Thanks Jiayi for your response. I am thinking on the same lines.  

Regarding using the same name and uuid, I believe the checkpoint state for an operator will be easy to identify if the uuid is the same as name.  But I am not sure if having a very long name and uuid or a character like parenthesis, etc might cause any issues, so just wanted to check.

Mans

On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao <[hidden email]> wrote:

Hi Mans!

Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1].

> Are there any restrictions on the length of the name and uuid attributes?

It’s pretty much the same as you define a string value, so there is no special restrictions on this.

> Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

I’m not a hundred percent sure about this but I run a testing program and it works fine.

> Can the name and uuid be the same ? 

Yes. But uuids accross operators cannot be same.

For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling.

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best,

Jiayi Liao

At 2019-11-16 18:35:38, "M Singh" <[hidden email]> wrote:

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans

 

[M Singh]

Hi Arvid:

Thanks for your clarification.

I am giving supplying uid for the stateful operators and find the following directory structure on in the chkpoint directory:

f4e78cb47f9dc12859558be7d15f39d0/chk-6/a4d87cda-2afd-47d4-8d3f-b0658466fb2d

The first part f4e78cb47f9dc12859558be7d15f39d0 is the job_id

Is there a way to map the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) -  to the uid assigned in the application ?

Thanks 

On Wednesday, November 20, 2019, 07:52:49 AM EST, Arvid Heise <[hidden email]> wrote:

Hi Mans,

just to follow up. There are no limitations for name or uuid.

The uuid will be in fact hashed internally while the StreamGraph is being generated, so all characters are allowed.

The name is only for debugging purposes and web ui. If you use very special characters, you may see oddities in logs/web ui, but nothing should break.

Spaces or parentheses should work in any case.

Best,

Arvid

On Sat, Nov 16, 2019 at 6:40 PM M Singh <[hidden email]> wrote:

Thanks Jiayi for your response. I am thinking on the same lines.  

Regarding using the same name and uuid, I believe the checkpoint state for an operator will be easy to identify if the uuid is the same as name.  But I am not sure if having a very long name and uuid or a character like parenthesis, etc might cause any issues, so just wanted to check.

Mans

On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao <[hidden email]> wrote:

Hi Mans!

Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1].

> Are there any restrictions on the length of the name and uuid attributes?

It’s pretty much the same as you define a string value, so there is no special restrictions on this.

> Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

I’m not a hundred percent sure about this but I run a testing program and it works fine.

> Can the name and uuid be the same ? 

Yes. But uuids accross operators cannot be same.

For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling.

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best,

Jiayi Liao

At 2019-11-16 18:35:38, "M Singh" <[hidden email]> wrote:

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans

 

[Congxian Qiu]

Hi

Currently, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) is generated by UUID.randomUUID(), so there is not a easy way to map this to the assigned in the application.

In another word, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) belongs to one checkpoint, and the assigned in the application belongs to one operator, they are different.

M Singh <[hidden email]> 于2019年11月21日周四 上午6:18写道：

Hi Arvid:

Thanks for your clarification.

I am giving supplying uid for the stateful operators and find the following directory structure on in the chkpoint directory:

f4e78cb47f9dc12859558be7d15f39d0/chk-6/a4d87cda-2afd-47d4-8d3f-b0658466fb2d

The first part f4e78cb47f9dc12859558be7d15f39d0 is the job_id

Is there a way to map the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) -  to the uid assigned in the application ?

Thanks 

On Wednesday, November 20, 2019, 07:52:49 AM EST, Arvid Heise <[hidden email]> wrote:

Hi Mans,

just to follow up. There are no limitations for name or uuid.

The uuid will be in fact hashed internally while the StreamGraph is being generated, so all characters are allowed.

The name is only for debugging purposes and web ui. If you use very special characters, you may see oddities in logs/web ui, but nothing should break.

Spaces or parentheses should work in any case.

Best,

Arvid

On Sat, Nov 16, 2019 at 6:40 PM M Singh <[hidden email]> wrote:

Thanks Jiayi for your response. I am thinking on the same lines.  

Regarding using the same name and uuid, I believe the checkpoint state for an operator will be easy to identify if the uuid is the same as name.  But I am not sure if having a very long name and uuid or a character like parenthesis, etc might cause any issues, so just wanted to check.

Mans

On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao <[hidden email]> wrote:

Hi Mans!

Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1].

> Are there any restrictions on the length of the name and uuid attributes?

It’s pretty much the same as you define a string value, so there is no special restrictions on this.

> Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

I’m not a hundred percent sure about this but I run a testing program and it works fine.

> Can the name and uuid be the same ? 

Yes. But uuids accross operators cannot be same.

For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling.

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best,

Jiayi Liao

At 2019-11-16 18:35:38, "M Singh" <[hidden email]> wrote:

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans

 

[M Singh]

Hi Congxian:

For my application i see many uuids under the chk-6 directory ( I posted one in the sample above).   I am trying to understand that if I restart the application with this checkpoint (which I believe I can just like a savepoint - I am using chk-6 as an example below)

1. I believe each chk-<d> - is a complete state of checkpoint. Is that correct ?

2. How to I point it to the checkpoint (chk-6) when I submit the job - Do I point it to the jobid or the chk-6 directory ?  I am presuming the latter (ie, pointing to the chk06 directory but just want to confirm.

3. Secondly, how does the application map the the files under the chk-6 to restore the state of each of the stateful operators ?

4. Is there any API by which I can examine the contents of the files under the chk-6 directory ?

Thanks for your help.

Mans

On Wednesday, November 20, 2019, 09:13:39 PM EST, Congxian Qiu <[hidden email]> wrote:

Hi

Currently, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) is generated by UUID.randomUUID(), so there is not a easy way to map this to the assigned in the application.

In another word, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) belongs to one checkpoint, and the assigned in the application belongs to one operator, they are different.

M Singh <[hidden email]> 于2019年11月21日周四 上午6:18写道：

Hi Arvid:

Thanks for your clarification.

I am giving supplying uid for the stateful operators and find the following directory structure on in the chkpoint directory:

f4e78cb47f9dc12859558be7d15f39d0/chk-6/a4d87cda-2afd-47d4-8d3f-b0658466fb2d

The first part f4e78cb47f9dc12859558be7d15f39d0 is the job_id

Is there a way to map the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) -  to the uid assigned in the application ?

Thanks 

On Wednesday, November 20, 2019, 07:52:49 AM EST, Arvid Heise <[hidden email]> wrote:

Hi Mans,

just to follow up. There are no limitations for name or uuid.

The uuid will be in fact hashed internally while the StreamGraph is being generated, so all characters are allowed.

The name is only for debugging purposes and web ui. If you use very special characters, you may see oddities in logs/web ui, but nothing should break.

Spaces or parentheses should work in any case.

Best,

Arvid

On Sat, Nov 16, 2019 at 6:40 PM M Singh <[hidden email]> wrote:

Thanks Jiayi for your response. I am thinking on the same lines.  

Regarding using the same name and uuid, I believe the checkpoint state for an operator will be easy to identify if the uuid is the same as name.  But I am not sure if having a very long name and uuid or a character like parenthesis, etc might cause any issues, so just wanted to check.

Mans

On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao <[hidden email]> wrote:

Hi Mans!

Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1].

> Are there any restrictions on the length of the name and uuid attributes?

It’s pretty much the same as you define a string value, so there is no special restrictions on this.

> Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

I’m not a hundred percent sure about this but I run a testing program and it works fine.

> Can the name and uuid be the same ? 

Yes. But uuids accross operators cannot be same.

For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling.

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best,

Jiayi Liao

At 2019-11-16 18:35:38, "M Singh" <[hidden email]> wrote:

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans

 

[Congxian Qiu]

Hi

1. I think this issue[1] can help to understand the directory layout

2. chk-6 directory or the metafilePath, for more information, you can ref to[2][3]

3. every checkpoint contains a meta-file record such, maybe you can ref to[4], and debug it for more information

4. currently, you need to go through  the restore logic to see if the files' contents

[1] https://issues.apache.org/jira/browse/FLINK-8531

[2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1153

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint

[4] https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointMetadataLoadingTest.java

M Singh <[hidden email]> 于2019年11月21日周四 下午7:44写道：

Hi Congxian:

For my application i see many uuids under the chk-6 directory ( I posted one in the sample above).   I am trying to understand that if I restart the application with this checkpoint (which I believe I can just like a savepoint - I am using chk-6 as an example below)

1. I believe each chk-<d> - is a complete state of checkpoint. Is that correct ?

2. How to I point it to the checkpoint (chk-6) when I submit the job - Do I point it to the jobid or the chk-6 directory ?  I am presuming the latter (ie, pointing to the chk06 directory but just want to confirm.

3. Secondly, how does the application map the the files under the chk-6 to restore the state of each of the stateful operators ?

4. Is there any API by which I can examine the contents of the files under the chk-6 directory ?

Thanks for your help.

Mans

On Wednesday, November 20, 2019, 09:13:39 PM EST, Congxian Qiu <[hidden email]> wrote:

Hi

Currently, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) is generated by UUID.randomUUID(), so there is not a easy way to map this to the assigned in the application.

In another word, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) belongs to one checkpoint, and the assigned in the application belongs to one operator, they are different.

Best,

Congxian

M Singh <[hidden email]> 于2019年11月21日周四 上午6:18写道：

Hi Arvid:

Thanks for your clarification.

I am giving supplying uid for the stateful operators and find the following directory structure on in the chkpoint directory:

f4e78cb47f9dc12859558be7d15f39d0/chk-6/a4d87cda-2afd-47d4-8d3f-b0658466fb2d

The first part f4e78cb47f9dc12859558be7d15f39d0 is the job_id

Is there a way to map the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) -  to the uid assigned in the application ?

Thanks 

On Wednesday, November 20, 2019, 07:52:49 AM EST, Arvid Heise <[hidden email]> wrote:

Hi Mans,

just to follow up. There are no limitations for name or uuid.

The uuid will be in fact hashed internally while the StreamGraph is being generated, so all characters are allowed.

The name is only for debugging purposes and web ui. If you use very special characters, you may see oddities in logs/web ui, but nothing should break.

Spaces or parentheses should work in any case.

Best,

Arvid

On Sat, Nov 16, 2019 at 6:40 PM M Singh <[hidden email]> wrote:

Thanks Jiayi for your response. I am thinking on the same lines.  

Regarding using the same name and uuid, I believe the checkpoint state for an operator will be easy to identify if the uuid is the same as name.  But I am not sure if having a very long name and uuid or a character like parenthesis, etc might cause any issues, so just wanted to check.

Mans

On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao <[hidden email]> wrote:

Hi Mans!

Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1].

> Are there any restrictions on the length of the name and uuid attributes?

It’s pretty much the same as you define a string value, so there is no special restrictions on this.

> Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

I’m not a hundred percent sure about this but I run a testing program and it works fine.

> Can the name and uuid be the same ? 

Yes. But uuids accross operators cannot be same.

For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling.

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best,

Jiayi Liao

At 2019-11-16 18:35:38, "M Singh" <[hidden email]> wrote:

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans

 

[M Singh]

Thanks so much Congxian for your pointers - I will try to go through them.  Mans

On Thursday, November 21, 2019, 07:26:49 AM EST, Congxian Qiu <[hidden email]> wrote:

Hi

1. I think this issue[1] can help to understand the directory layout

2. chk-6 directory or the metafilePath, for more information, you can ref to[2][3]

3. every checkpoint contains a meta-file record such, maybe you can ref to[4], and debug it for more information

4. currently, you need to go through  the restore logic to see if the files' contents

[1] https://issues.apache.org/jira/browse/FLINK-8531

[2] https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1153

[3] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/state/checkpoints.html#resuming-from-a-retained-checkpoint

[4] https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointMetadataLoadingTest.java

M Singh <[hidden email]> 于2019年11月21日周四 下午7:44写道：

Hi Congxian:

For my application i see many uuids under the chk-6 directory ( I posted one in the sample above).   I am trying to understand that if I restart the application with this checkpoint (which I believe I can just like a savepoint - I am using chk-6 as an example below)

1. I believe each chk-<d> - is a complete state of checkpoint. Is that correct ?

2. How to I point it to the checkpoint (chk-6) when I submit the job - Do I point it to the jobid or the chk-6 directory ?  I am presuming the latter (ie, pointing to the chk06 directory but just want to confirm.

3. Secondly, how does the application map the the files under the chk-6 to restore the state of each of the stateful operators ?

4. Is there any API by which I can examine the contents of the files under the chk-6 directory ?

Thanks for your help.

Mans

On Wednesday, November 20, 2019, 09:13:39 PM EST, Congxian Qiu <[hidden email]> wrote:

Hi

Currently, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) is generated by UUID.randomUUID(), so there is not a easy way to map this to the assigned in the application.

In another word, the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) belongs to one checkpoint, and the assigned in the application belongs to one operator, they are different.

Best,

Congxian

M Singh <[hidden email]> 于2019年11月21日周四 上午6:18写道：

Hi Arvid:

Thanks for your clarification.

I am giving supplying uid for the stateful operators and find the following directory structure on in the chkpoint directory:

f4e78cb47f9dc12859558be7d15f39d0/chk-6/a4d87cda-2afd-47d4-8d3f-b0658466fb2d

The first part f4e78cb47f9dc12859558be7d15f39d0 is the job_id

Is there a way to map the last part (uuid a4d87cda-2afd-47d4-8d3f-b0658466fb2d) -  to the uid assigned in the application ?

Thanks 

On Wednesday, November 20, 2019, 07:52:49 AM EST, Arvid Heise <[hidden email]> wrote:

Hi Mans,

just to follow up. There are no limitations for name or uuid.

The uuid will be in fact hashed internally while the StreamGraph is being generated, so all characters are allowed.

The name is only for debugging purposes and web ui. If you use very special characters, you may see oddities in logs/web ui, but nothing should break.

Spaces or parentheses should work in any case.

Best,

Arvid

On Sat, Nov 16, 2019 at 6:40 PM M Singh <[hidden email]> wrote:

Thanks Jiayi for your response. I am thinking on the same lines.  

Regarding using the same name and uuid, I believe the checkpoint state for an operator will be easy to identify if the uuid is the same as name.  But I am not sure if having a very long name and uuid or a character like parenthesis, etc might cause any issues, so just wanted to check.

Mans

On Saturday, November 16, 2019, 11:19:08 AM EST, Jiayi Liao <[hidden email]> wrote:

Hi Mans!

Firstly let’s see how operator’s name and uid is used. AFAIK, operator’s name is used in WebUI and metrics reporting, and uid is used to mark the uniqueness of operator which is useful when you’re using state[1].

> Are there any restrictions on the length of the name and uuid attributes?

It’s pretty much the same as you define a string value, so there is no special restrictions on this.

> Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

I’m not a hundred percent sure about this but I run a testing program and it works fine.

> Can the name and uuid be the same ? 

Yes. But uuids accross operators cannot be same.

For me I usually set name and uuid for almost every operator, which gives me better experience in monitoring and scaling.

Hope this helps.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#matching-operator-state

Best,

Jiayi Liao

At 2019-11-16 18:35:38, "M Singh" <[hidden email]> wrote:

Hi:

I am working on a project and wanted to find out what are the best practices for setting name and uuid for operators:

1. Are there any restrictions on the length of the name and uuid attributes ?

2. Are there any restrictions on the characters used for name and uuid (blank spaces, etc) ?

3. Can the name and uuid be the same ?

Please let me know if there is any other advice.

Thanks

Mans