AW: Re: Lot of RocksDB files in tmp Directory

[Dominique Rondé-2]

First of all, thanks for the explanation. That sounds reasonable.

But I started the flink routes 3 days ago and went out for the weekend. Since we are two people with access to flink i guess there is something strange happen...

Greets

Dominique 

Von meinem Samsung Gerät gesendet.

-------- Ursprüngliche Nachricht --------
Von: "xiaogang.sxg" <[hidden email]>
Datum: 07.11.16 04:00 (GMT+01:00)
An: [hidden email]
Betreff: Re: Lot of RocksDB files in tmp Directory

RocksDB is orginally  implemented in C. To use it in Java, Flink uses a RocksDB jar with JNI techniques. 

Whenever a Flink job starts, the RocksDB libraries will be loaded. The file named librocksdb*.so will be extracted from the RocksDB jar and will be put in the tmp directory. So I think you have run Flink jobs many many times in the three days to get those 451 files.

I don’t think the numerous files are a big problem because they are located in the tmp directory and  they can be automatically cleaned by OS.

在 2016年11月7日，上午8:24，Dominique Rondé <[hidden email]> 写道：

Hi @ll,

we just change the backend from filesystem to RocksDB. Since that time (3 Days) we got 451 files with 1.8 GB stored in the tmp-Directory. All files are named librocksdb*.so

Did we something wrong or is it a Bug?

Greets

Dominique 

Von meinem Samsung Gerät gesendet.

[Aljoscha Krettek]

Hi,

could it be that the job has restarted due to failures a large number of times?

Cheers,

Aljoscha

On Mon, 7 Nov 2016 at 09:21 Dominique Rondé <[hidden email]> wrote:

First of all, thanks for the explanation. That sounds reasonable.

But I started the flink routes 3 days ago and went out for the weekend. Since we are two people with access to flink i guess there is something strange happen...

Greets

Dominique 

Von meinem Samsung Gerät gesendet.

-------- Ursprüngliche Nachricht --------
Von: "xiaogang.sxg" <[hidden email]>
Datum: 07.11.16 04:00 (GMT+01:00)
An: [hidden email]
Betreff: Re: Lot of RocksDB files in tmp Directory

RocksDB is orginally  implemented in C. To use it in Java, Flink uses a RocksDB jar with JNI techniques. 

Whenever a Flink job starts, the RocksDB libraries will be loaded. The file named librocksdb*.so will be extracted from the RocksDB jar and will be put in the tmp directory. So I think you have run Flink jobs many many times in the three days to get those 451 files.

I don’t think the numerous files are a big problem because they are located in the tmp directory and  they can be automatically cleaned by OS.

在 2016年11月7日，上午8:24，Dominique Rondé <[hidden email]> 写道：

Hi @ll,

we just change the backend from filesystem to RocksDB. Since that time (3 Days) we got 451 files with 1.8 GB stored in the tmp-Directory. All files are named librocksdb*.so

Did we something wrong or is it a Bug?

Greets

Dominique 

Von meinem Samsung Gerät gesendet.

[Dominique Rondé-2]

Hi Aljoscha,

i try to reproduce this. The job is started only once and fill my /tmp Directory within minutes.

Greets

Dominique

Am 14.11.2016 um 16:31 schrieb Aljoscha Krettek:

Hi,

could it be that the job has restarted due to failures a large number of times?

Cheers,

Aljoscha

On Mon, 7 Nov 2016 at 09:21 Dominique Rondé <[hidden email]> wrote:

First of all, thanks for the explanation. That sounds reasonable.

But I started the flink routes 3 days ago and went out for the weekend. Since we are two people with access to flink i guess there is something strange happen...

Greets

Dominique 

Von meinem Samsung Gerät gesendet.

-------- Ursprüngliche Nachricht --------
Von: "xiaogang.sxg" <[hidden email]>
Datum: 07.11.16 04:00 (GMT+01:00)
An: [hidden email]
Betreff: Re: Lot of RocksDB files in tmp Directory

RocksDB is orginally  implemented in C. To use it in Java, Flink uses a RocksDB jar with JNI techniques. 

Whenever a Flink job starts, the RocksDB libraries will be loaded. The file named librocksdb*.so will be extracted from the RocksDB jar and will be put in the tmp directory. So I think you have run Flink jobs many many times in the three days to get those 451 files.

I don’t think the numerous files are a big problem because they are located in the tmp directory and  they can be automatically cleaned by OS.

在 2016年11月7日，上午8:24，Dominique Rondé <[hidden email]> 写道：

Hi @ll,

we just change the backend from filesystem to RocksDB. Since that time (3 Days) we got 451 files with 1.8 GB stored in the tmp-Directory. All files are named librocksdb*.so

Did we something wrong or is it a Bug?

Greets

Dominique 

Von meinem Samsung Gerät gesendet.

[Aljoscha Krettek]

Ok, please let me know when you manage to reproduce.

Cheers,

Aljoscha

On Tue, 15 Nov 2016 at 15:00 Dominique Rondé <[hidden email]> wrote:

Hi Aljoscha,

i try to reproduce this. The job is started only once and fill my /tmp Directory within minutes.

Greets

Dominique

Am 14.11.2016 um 16:31 schrieb Aljoscha Krettek:

Hi,

could it be that the job has restarted due to failures a large number of times?

Cheers,

Aljoscha

On Mon, 7 Nov 2016 at 09:21 Dominique Rondé <[hidden email]> wrote:

First of all, thanks for the explanation. That sounds reasonable.

But I started the flink routes 3 days ago and went out for the weekend. Since we are two people with access to flink i guess there is something strange happen...

Greets

Dominique 

Von meinem Samsung Gerät gesendet.

-------- Ursprüngliche Nachricht --------
Von: "xiaogang.sxg" <[hidden email]>
Datum: 07.11.16 04:00 (GMT+01:00)
An: [hidden email]
Betreff: Re: Lot of RocksDB files in tmp Directory

RocksDB is orginally  implemented in C. To use it in Java, Flink uses a RocksDB jar with JNI techniques. 

Whenever a Flink job starts, the RocksDB libraries will be loaded. The file named librocksdb*.so will be extracted from the RocksDB jar and will be put in the tmp directory. So I think you have run Flink jobs many many times in the three days to get those 451 files.

I don’t think the numerous files are a big problem because they are located in the tmp directory and  they can be automatically cleaned by OS.

在 2016年11月7日，上午8:24，Dominique Rondé <[hidden email]> 写道：

Hi @ll,

we just change the backend from filesystem to RocksDB. Since that time (3 Days) we got 451 files with 1.8 GB stored in the tmp-Directory. All files are named librocksdb*.so

Did we something wrong or is it a Bug?

Greets

Dominique 

Von meinem Samsung Gerät gesendet.