Re: Blobstorage Locally and on HDFS

Posted by snntr on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Blobstorage-Locally-and-on-HDFS-tp9262p9324.html

Hi Ufuk,

any ideas? Any configuration that could be wrong?

Cheers,

Konstantin

On 30.09.2016 13:13, Konstantin Knauf wrote:

> Hi Ufuk,
>
> thanks for your quick answer.
>
> Setup: 2 Servers, each running a JM as well as TM
>
> 1) Removing all existing blobstores locally (/tmp) as well as on HDFS
> 2) Starting a flink streaming job
>
> Now there are the following BLOBs:
>
> Local:
>
> *Leader JM:
>
> 4.0K    /tmp/blobStore-563a8820-9617-4d89-97a7-fc3cc258dff4/incoming
>
> 64M     /tmp/blobStore-563a8820-9617-4d89-97a7-fc3cc258dff4
>
> 64M     /tmp/blobStore-563a8820-9617-4d89-97a7-fc3cc258dff4/cache
>
> 64M     /tmp/blobStore-c6b93d41-8916-4a8d-b595-6e35f0b10401
>
> 64M     /tmp/blobStore-c6b93d41-8916-4a8d-b595-6e35f0b10401/cache
>
> *Standby JM:
>
> 64M     /tmp/blobStore-4cbfd3c0-2a70-4485-8fc0-045ca7f08cea
>
> 64M     /tmp/blobStore-4cbfd3c0-2a70-4485-8fc0-045ca7f08cea/cache
>
> HDFS:
>
> 66595700 2016-09-30 13:03
> <..>/flink/blob/cache/blob_da76e12b949a83404f97b6eb59416deaa31a907b
>
>
> 3) Cancelinng both jobs via command line:
>
> Now there are the following BLOBs:
>
> **same as above**
>
> When starting the same job again, no new blobs are created.
>
> Is it a problem to delete local blobStores of running jobs or will the
> blobs just be downloaded again from HDFS if needed?
>
> Cheers,
>
> Konstantin
>
>
>
> Is it correct, that ea
>
> On 30.09.2016 10:28, Ufuk Celebi wrote:
>> On Fri, Sep 30, 2016 at 9:12 AM, Konstantin Knauf
>> <[hidden email]> wrote:
>>> we are running a Flink (1.1.2) Stand-Alone Cluster with JM HA, and HDFS
>>> as checkpoint and recovery storage dir. What we see is that blobStores
>>> are stored in HDFS as well as under the local Jobmanagers and
>>> Taskmanagers /tmp directory.
>>>
>>> Is this the expected behaviour? Is there any documentation on which
>>> blobs are stored locally and which are stored in HDFS in our case? In
>>> particular, we would need to know when it is save to delete blobs stored
>>> locally because there are not cleanup up by Flink and fill up the /tmp
>>> partition eventually.
>>
>> BLOBs are copied to another directory in case of HA in order to be
>> available for other job managers that might take over.
>>
>> On regular termination (cancel, finish) all BLOBs should be cleaned
>> up. With hard failures, it can happen that BLOBs are not cleaned up.
>>
>> Do you know in which cases you see BLOBs not being cleaned up? If it
>> is the first one, that sounds like a bug to me.
>>
>> – Ufuk
>>
>

--
Konstantin Knauf * [hidden email] * [phone]
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082