Some question about flink temp files

[Reo Lei]

Hi all,

Recently, I found flink tmp files(localState, blobStore-*, flink-dist-cach-*, flink-io-*, flink-netty-shuffle-* etc.) has been grown to a total of about 6GB size. I have no idea what usage abot this files, and How big it will grow. 

So, I have some questions about the tmep files of flink as follow:

1, what is the usage of this tmp files?

2, Is there have any mechanism of flink to manage this files?

3, Have any situation I need to delete the useless files manually？

4, I know I can specify the temp files path by "io.tmp.dirs" config for flink cluster. But when I use LocalEnvironment, how can I config this path instead of use the default path("/tmp")?

Thanks.

BR,

Reo

[r_khachatryan]

Hi Reo,

Please find the answers to your questions below.

> 1, what is the usage of this tmp files?
These files are used by Flink internally for things like caching state locally, storing jars and so on. They are not intended for the end-user.
> 2, Is there have any mechanism of flink to manage this files?
Yes, they are created and deleted by Flink when not used.
> 3, Have any situation I need to delete the useless files manually？
No.
> 4, I know I can specify the temp files path by "io.tmp.dirs" config for flink cluster. But when I use LocalEnvironment, how can I config this path instead of use the default path("/tmp")?
Please use "java.io.tmpdir" *system property* for standalone deployment [1].

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html

On Fri, Mar 20, 2020 at 11:55 AM Reo Lei <[hidden email]> wrote:

Hi all,

Recently, I found flink tmp files(localState, blobStore-*, flink-dist-cach-*, flink-io-*, flink-netty-shuffle-* etc.) has been grown to a total of about 6GB size. I have no idea what usage abot this files, and How big it will grow. 

So, I have some questions about the tmep files of flink as follow:

1, what is the usage of this tmp files?

2, Is there have any mechanism of flink to manage this files?

3, Have any situation I need to delete the useless files manually？

4, I know I can specify the temp files path by "io.tmp.dirs" config for flink cluster. But when I use LocalEnvironment, how can I config this path instead of use the default path("/tmp")?

Thanks.

BR,

Reo

[Reo Lei]

Thank you  Roman!

That is very helpful! Thank you!

BR,

Reo

Khachatryan Roman <[hidden email]> 于2020年3月20日周五 下午11:13写道：

Hi Reo,

Please find the answers to your questions below.

> 1, what is the usage of this tmp files?
These files are used by Flink internally for things like caching state locally, storing jars and so on. They are not intended for the end-user.
> 2, Is there have any mechanism of flink to manage this files?
Yes, they are created and deleted by Flink when not used.
> 3, Have any situation I need to delete the useless files manually？
No.
> 4, I know I can specify the temp files path by "io.tmp.dirs" config for flink cluster. But when I use LocalEnvironment, how can I config this path instead of use the default path("/tmp")?
Please use "java.io.tmpdir" *system property* for standalone deployment [1].

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html

Regards,
Roman

On Fri, Mar 20, 2020 at 11:55 AM Reo Lei <[hidden email]> wrote:

Hi all,

Recently, I found flink tmp files(localState, blobStore-*, flink-dist-cach-*, flink-io-*, flink-netty-shuffle-* etc.) has been grown to a total of about 6GB size. I have no idea what usage abot this files, and How big it will grow. 

So, I have some questions about the tmep files of flink as follow:

1, what is the usage of this tmp files?

2, Is there have any mechanism of flink to manage this files?

3, Have any situation I need to delete the useless files manually？

4, I know I can specify the temp files path by "io.tmp.dirs" config for flink cluster. But when I use LocalEnvironment, how can I config this path instead of use the default path("/tmp")?

Thanks.

BR,

Reo