How to make Flink to write less temporary files?
Locked
How to make Flink to write less temporary files?
 
|
Hi, I’m having the problem that my tmp dir is running out of hard drive space when running a map reduce job on a 1TB file. The job fails with "no space left on device“ exception. Probably the intermediate result set is getting too big. Is there a way to avoid this? Or make flink to write less temporary files? Thanks, Malte |
Re: How to make Flink to write less temporary files?
|
|
Hey Malte, thanks for reporting the issue. Did you change the default configuration? I've just checked and the default config hard codes the heap size for the task managers to 512 MB. In that case some of the algorithms will start spilling to disk earlier than necessary (assuming that you have more main memory available). You can either remove the config key "taskmanager.heap.mb" and let the JVM set a default max heap size or set the config key to a value appropriate for your machines/setup. Could you try this out and report back? Regarding the temp directories: How much disk space is available before running your program? I remember a user reporting the same issue, because of some old files lingering around in the temp directories. – Ufuk On Sun, Nov 9, 2014 at 10:54 PM, Malte Schwarzer <[hidden email]> wrote:
|
Re: How to make Flink to write less temporary files?
|
|
I also see a lot of blobStore-XXX directories in /tmp
On Mon, Nov 10, 2014 at 8:48 AM, Ufuk Celebi <[hidden email]> wrote:
|
Re: How to make Flink to write less temporary files?
|
|
@Flavio: These are used by the blob manager to distribute the JAR files in the cluster. I thought they were deleted when the task managers shut down. Maybe Till or Daniel can provide some information on the life cycle of blob manager objects. On Mon, Nov 10, 2014 at 9:24 AM, Flavio Pompermaier <[hidden email]> wrote:
|
Re: How to make Flink to write less temporary files?
|
|
Hi Flavio,
the BlobManager is supposed to delete the blobStore directories either when the job finishes or when the cluster is shutdown. However, if the system crashes, then it won't clean up the directories. Has your system crashed recently and thus has not been shut down properly? On Mon, Nov 10, 2014 at 9:37 AM, Ufuk Celebi <[hidden email]> wrote: > @Flavio: These are used by the blob manager to distribute the JAR files in > the cluster. I thought they were deleted when the task managers shut down. > Maybe Till or Daniel can provide some information on the life cycle of blob > manager objects. > > On Mon, Nov 10, 2014 at 9:24 AM, Flavio Pompermaier <[hidden email]> > wrote: >> >> I also see a lot of blobStore-XXX directories in /tmp >> >> >> On Mon, Nov 10, 2014 at 8:48 AM, Ufuk Celebi <[hidden email]> wrote: >>> >>> Hey Malte, >>> >>> thanks for reporting the issue. Did you change the default configuration? >>> I've just checked and the default config hard codes the heap size for the >>> task managers to 512 MB. In that case some of the algorithms will start >>> spilling to disk earlier than necessary (assuming that you have more main >>> memory available). You can either remove the config key >>> "taskmanager.heap.mb" and let the JVM set a default max heap size or set the >>> config key to a value appropriate for your machines/setup. Could you try >>> this out and report back? >>> >>> Regarding the temp directories: How much disk space is available before >>> running your program? I remember a user reporting the same issue, because of >>> some old files lingering around in the temp directories. >>> >>> – Ufuk >>> >>> >>> On Sun, Nov 9, 2014 at 10:54 PM, Malte Schwarzer <[hidden email]> wrote: >>>> >>>> Hi, >>>> >>>> I’m having the problem that my tmp dir is running out of hard drive >>>> space when running a map reduce job on a 1TB file. The job fails with "no >>>> space left on device“ exception. >>>> >>>> Probably the intermediate result set is getting too big. Is there a way >>>> to avoid this? Or make flink to write less temporary files? >>>> >>>> Thanks, >>>> Malte >>>> >>>> >>> >> >> > |
Re: How to make Flink to write less temporary files?
|
|
I would assume that the blobStore fields are rather small (they are only jar files so far). I would look for *.channel files, which are spilled intermediate results. They can get pretty large for large jobs. |
Re: How to make Flink to write less temporary files?
|
|
My blobStore fileds are small, but each *.channel file is around 170MB. Before I start by Flink job I’ve 25GB free space available in my tmp-dir and my taskmanager heap size is currently at 24GB. I’m using a cluster with 10 nodes. Is this enough space to process a 1TB file? Von: Stephan Ewen <[hidden email]> Antworten an: <[hidden email]> Datum: Montag, 10. November 2014 10:35 An: <[hidden email]> Betreff: Re: How to make Flink to write less temporary files? I would assume that the blobStore fields are rather small (they are only jar files so far). I would look for *.channel files, which are spilled intermediate results. They can get pretty large for large jobs. |
Re: How to make Flink to write less temporary files?
|
|
Hi! With 10 nodes and 25 GB on each node, you have 250 GB space to spill temporary files. You also seem to have roughly the same size in JVM Heap, out of which Flink can use roughly 2/3. When you process 1 TB, 250 GB JVM heap and 250 GB temp file space may not be enough, it is less than the initial data size. I think you need simply need more disk space for a job like that... Stephan On Mon, Nov 10, 2014 at 10:54 AM, Malte Schwarzer <[hidden email]> wrote:
|
|
|
Hi, maybe the tmp-dir's are mounted as ramdisks? Some Linux distributions have this as a default configuration. On Mon, Nov 10, 2014 at 2:22 AM, Stephan Ewen <[hidden email]> wrote:
|
Re: How to make Flink to write less temporary files?
|
|
In reply to this post by Stephan Ewen
What's the estimated amount of disk space for such a job? Or how can I calculate it? Malte Von: Stephan Ewen <[hidden email]> Antworten an: <[hidden email]> Datum: Montag, 10. November 2014 11:22 An: <[hidden email]> Betreff: Re: How to make Flink to write less temporary files? Hi! With 10 nodes and 25 GB on each node, you have 250 GB space to spill temporary files. You also seem to have roughly the same size in JVM Heap, out of which Flink can use roughly 2/3. When you process 1 TB, 250 GB JVM heap and 250 GB temp file space may not be enough, it is less than the initial data size. I think you need simply need more disk space for a job like that... Stephan On Mon, Nov 10, 2014 at 10:54 AM, Malte Schwarzer <[hidden email]> wrote:
|
Re: How to make Flink to write less temporary files?
|
|
It is hard to calculate, it very much depends on the job: - Is it running filters that reduce the data volume early? - It it possibly running operations that blow up the size of an intermediate result? I would in general assume to use as much temp space than the input data size, unless you have a lot of RAM such that the system can process the job completely in memory. Unless you immediately filter the input data aggressively (you have a highly selective filter function after the readTextFile(...) or so). Stephan On Mon, Nov 10, 2014 at 5:59 PM, Malte Schwarzer <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |