Spill RocksDB to external Storage

[Rex Fenley]

Hi,

I'm wondering if the RocksDB state storage has any way of automatically spilling into S3 (assuming that's what's set for checkpoints) when the host disk starts filling up too much? If not, is the expectation that the host disk must always carry all the relevant data in RocksDB, meaning we're bound by the size of host storage?

Thanks!

--

[David Anderson-4]

RocksDB can not be configured to spill to another filesystem or object store. It is designed as an embedded database, and each task manager needs to have sufficient disk space for its state on the host disk. You might be tempted to use network attached storage for the working state, but that's usually a bad idea (for performance reasons).

Regards,

David

On Fri, Dec 11, 2020 at 10:59 PM Rex Fenley <[hidden email]> wrote:

Hi,

I'm wondering if the RocksDB state storage has any way of automatically spilling into S3 (assuming that's what's set for checkpoints) when the host disk starts filling up too much? If not, is the expectation that the host disk must always carry all the relevant data in RocksDB, meaning we're bound by the size of host storage?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

[Rex Fenley]

Noted, thanks!

On Sat, Dec 12, 2020 at 2:28 AM David Anderson <[hidden email]> wrote:

RocksDB can not be configured to spill to another filesystem or object store. It is designed as an embedded database, and each task manager needs to have sufficient disk space for its state on the host disk. You might be tempted to use network attached storage for the working state, but that's usually a bad idea (for performance reasons).

Regards,

David

On Fri, Dec 11, 2020 at 10:59 PM Rex Fenley <[hidden email]> wrote:

Hi,

I'm wondering if the RocksDB state storage has any way of automatically spilling into S3 (assuming that's what's set for checkpoints) when the host disk starts filling up too much? If not, is the expectation that the host disk must always carry all the relevant data in RocksDB, meaning we're bound by the size of host storage?

Thanks!

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--