about blob.storage.dir and .buffer files

[Gwenhael Pasquiers]

Hello,

 

We got a question about blob.storage.dir and it’s .buffer files :

 

What are they ? And are they cleaned or is there a way to limit their size and to evaluate the necessary space ?

We got a node root volume disk filled by those files (~20GB) and it crashed.

 

Well, the root was filled because we changed the path to /tmp_flink and it was on the root filesystem, our bad. We changed it in an urge because the default path (/tmp) was being periodically cleaned by the OS and that made flink crash.

 

B.R.

 

[Till Rohrmann]

Hi Gwenhael,

in theory the blob storage files can be any binary data. At the moment, this is however only used to distribute the user code jars. The jars are kept around as long as the job is running. Every library-cache-manager.cleanup.interval interval the files are checked and those which are no longer referenced are deleted. In the case of a termination of Flink all files should be purged. If this is not the case, then we have to check what the problem is.

Could you check the size of your user jars?

Cheers,
Till

​

On Wed, Jan 27, 2016 at 4:38 PM, Gwenhael Pasquiers <[hidden email]> wrote:

Hello,

 

We got a question about blob.storage.dir and it’s .buffer files :

 

What are they ? And are they cleaned or is there a way to limit their size and to evaluate the necessary space ?

We got a node root volume disk filled by those files (~20GB) and it crashed.

 

Well, the root was filled because we changed the path to /tmp_flink and it was on the root filesystem, our bad. We changed it in an urge because the default path (/tmp) was being periodically cleaned by the OS and that made flink crash.

 

B.R.

 

[Stephan Ewen]

Hi Gwenhael!

Let's look into this and fix anything we find. Can you briefly tell us:

  - How much data is in the blob-store directory, versus in the buffer files?

  - How many buffer files do you have and how large are they in average?

Greetings,

Stephan

On Thu, Jan 28, 2016 at 10:18 AM, Till Rohrmann <[hidden email]> wrote:

Hi Gwenhael,

in theory the blob storage files can be any binary data. At the moment, this is however only used to distribute the user code jars. The jars are kept around as long as the job is running. Every library-cache-manager.cleanup.interval interval the files are checked and those which are no longer referenced are deleted. In the case of a termination of Flink all files should be purged. If this is not the case, then we have to check what the problem is.

Could you check the size of your user jars?

Cheers,
Till

​

On Wed, Jan 27, 2016 at 4:38 PM, Gwenhael Pasquiers <[hidden email]> wrote:

Hello,

 

We got a question about blob.storage.dir and it’s .buffer files :

 

What are they ? And are they cleaned or is there a way to limit their size and to evaluate the necessary space ?

We got a node root volume disk filled by those files (~20GB) and it crashed.

 

Well, the root was filled because we changed the path to /tmp_flink and it was on the root filesystem, our bad. We changed it in an urge because the default path (/tmp) was being periodically cleaned by the OS and that made flink crash.

 

B.R.

 

[Gwenhael Pasquiers]

Hi !

 

Here are the answers :

  - How much data is in the blob-store directory, versus in the buffer files? :

   around 20MB  per application  versus 20gb ( one specific windowing app may be).

 

  - How many buffer files do you have and how large are they in average? :

standard apps ( no buffer files or really small ones) , the one app : around 10 files of 2gb each in (ex /tmp_flink/flink-io-*/b3177f8e1910a686ccb48c1da533581d06305acccc2a370734b493327b117757.1441.buffer)

 

 

 

 

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Stephan Ewen
Sent: jeudi 28 janvier 2016 15:35
To: [hidden email]
Subject: Re: about blob.storage.dir and .buffer files

 

Hi Gwenhael!

 

Let's look into this and fix anything we find. Can you briefly tell us:

 

  - How much data is in the blob-store directory, versus in the buffer files?

 

  - How many buffer files do you have and how large are they in average?

 

Greetings,

Stephan

 

 

 

 

On Thu, Jan 28, 2016 at 10:18 AM, Till Rohrmann <[hidden email]> wrote:

Hi Gwenhael,

in theory the blob storage files can be any binary data. At the moment, this is however only used to distribute the user code jars. The jars are kept around as long as the job is running. Every library-cache-manager.cleanup.interval interval the files are checked and those which are no longer referenced are deleted. In the case of a termination of Flink all files should be purged. If this is not the case, then we have to check what the problem is.

Could you check the size of your user jars?

Cheers,
Till

​

 

On Wed, Jan 27, 2016 at 4:38 PM, Gwenhael Pasquiers <[hidden email]> wrote:

Hello,

 

We got a question about blob.storage.dir and it’s .buffer files :

 

What are they ? And are they cleaned or is there a way to limit their size and to evaluate the necessary space ?

We got a node root volume disk filled by those files (~20GB) and it crashed.

 

Well, the root was filled because we changed the path to /tmp_flink and it was on the root filesystem, our bad. We changed it in an urge because the default path (/tmp) was being periodically cleaned by the OS and that made flink crash.

 

B.R.