Disk full problem faced due to the Flink tmp directory contents

[Papadopoulos, Konstantinos]

Hi all,

 

We are developing several batch processing applications using the DataSet API of the Apache Flink.

For the time being, we are facing an issue with one of our production environments since its disk usage increase enormously. After a quick investigation, we concluded that the /tmp/flink-io-{} directory (under the parent directory of the Apache Flink deployment) contains files of more than 1TB and we need to regularly delete them in order to return our system to its proper functionality. On the first sight, there is no significant impact when deleting these temp files. So, I need your help to answer the following questions:

• What kind of data does it stored to the aforementioned directory?
• Why does the respective files have such an enormous size?
• How can we limit the size of the data written to the respective directory?
• Is there any way  to delete such files automatically when not needed yet?

 

Thanks in advance for your help,

Konstantinos

 

[Ken Krugler]

Hi Konstantinos,

Typically the data that you are seeing is from records being spilled to disk during groupBy/join operations, where the size of one (or multiple, for the join case) data sets exceeds what will fit in memory.

And yes, these files can get big, e.g. as big as the sum of your input data sizes.

If you split your data stream (one data set being processed by multiple operators) then the summed temp size can be multiplied.

You can specify multiple disks to use as temp directories (comma-separated list in Flink config), so that’s one way to avoid a single disk becoming too full.

You can take a single workflow and break it into multiple pieces that you run sequentially, as that can reduce the high water mark for total spilled files.

You can write intermediate results to a file, versus relying on spills. Though if you use HDFS, and HDFS is using the same disks in your cluster, that obviously won’t help, and in fact can be worse due to replication of data.

As far as auto-deletion goes, I don’t think Flink supports this. In our case, after a job has run we can a shell script (via ssh) on slaves to remove temp files.

— Ken

PS - note that logging can also chew up a lot of space, if you set the log level to DEBUG, due to HTTP wire traffic. 

On Jul 10, 2019, at 3:51 AM, Papadopoulos, Konstantinos <[hidden email]> wrote:

Hi all,

 

We are developing several batch processing applications using the DataSet API of the Apache Flink.

For the time being, we are facing an issue with one of our production environments since its disk usage increase enormously. After a quick investigation, we concluded that the /tmp/flink-io-{} directory (under the parent directory of the Apache Flink deployment) contains files of more than 1TB and we need to regularly delete them in order to return our system to its proper functionality. On the first sight, there is no significant impact when deleting these temp files. So, I need your help to answer the following questions:

• What kind of data does it stored to the aforementioned directory?
• Why does the respective files have such an enormous size?
• How can we limit the size of the data written to the respective directory?
• Is there any way  to delete such files automatically when not needed yet?

 

Thanks in advance for your help,

Konstantinos

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra

[Fabian Hueske-2]

Hi,

AFAIK Flink should remove temporary files automatically when they are not needed anymore.

However, I'm not 100% sure that there are not corner cases when a TM crashes.

In general it is a good idea to properly configure the directories that Flink uses for spilling, logging, blob storage, etc.

As Ken said, you can provide multiple directories to write temp data to multiple disks (using the combined IO rate).

Best, Fabian

Am Mi., 10. Juli 2019 um 15:16 Uhr schrieb Ken Krugler <[hidden email]>:

Hi Konstantinos,

Typically the data that you are seeing is from records being spilled to disk during groupBy/join operations, where the size of one (or multiple, for the join case) data sets exceeds what will fit in memory.

And yes, these files can get big, e.g. as big as the sum of your input data sizes.

If you split your data stream (one data set being processed by multiple operators) then the summed temp size can be multiplied.

You can specify multiple disks to use as temp directories (comma-separated list in Flink config), so that’s one way to avoid a single disk becoming too full.

You can take a single workflow and break it into multiple pieces that you run sequentially, as that can reduce the high water mark for total spilled files.

You can write intermediate results to a file, versus relying on spills. Though if you use HDFS, and HDFS is using the same disks in your cluster, that obviously won’t help, and in fact can be worse due to replication of data.

As far as auto-deletion goes, I don’t think Flink supports this. In our case, after a job has run we can a shell script (via ssh) on slaves to remove temp files.

— Ken

PS - note that logging can also chew up a lot of space, if you set the log level to DEBUG, due to HTTP wire traffic. 

On Jul 10, 2019, at 3:51 AM, Papadopoulos, Konstantinos <[hidden email]> wrote:

Hi all,

 

We are developing several batch processing applications using the DataSet API of the Apache Flink.

For the time being, we are facing an issue with one of our production environments since its disk usage increase enormously. After a quick investigation, we concluded that the /tmp/flink-io-{} directory (under the parent directory of the Apache Flink deployment) contains files of more than 1TB and we need to regularly delete them in order to return our system to its proper functionality. On the first sight, there is no significant impact when deleting these temp files. So, I need your help to answer the following questions:

• What kind of data does it stored to the aforementioned directory?
• Why does the respective files have such an enormous size?
• How can we limit the size of the data written to the respective directory?
• Is there any way  to delete such files automatically when not needed yet?

 

Thanks in advance for your help,

Konstantinos

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra