Migrating from one state backend to another

[Josh]

I'm trying to decide whether to set-up RocksDB now or later.

The state for my Flink jobs right now will be able to fit into memory, so I can use the filesystem state backend. In a few months time it is likely the state will no longer fit into memory, so I will want to use the RocksDB backend.

I was just wondering if it's possible/easy to use savepoints to migrate existing state from the filesystem backend to the RocksDB backend? As I would not want to lose any job state when switching to RocksDB. If there's a way to do it then I can worry about RocksDB later.

Thanks!

Josh

[Aljoscha Krettek]

Hi,

right now migrating from one state backend to another is not possible. I have it in the back of my head, however, that we should introduce a common serialized representation of state to make this possible in the future. (Both for checkpoints and savepoints, which use the same mechanism underneath.)

Cheers,

Aljoscha

On Wed, 15 Jun 2016 at 00:04 Josh <[hidden email]> wrote:

I'm trying to decide whether to set-up RocksDB now or later.

The state for my Flink jobs right now will be able to fit into memory, so I can use the filesystem state backend. In a few months time it is likely the state will no longer fit into memory, so I will want to use the RocksDB backend.

I was just wondering if it's possible/easy to use savepoints to migrate existing state from the filesystem backend to the RocksDB backend? As I would not want to lose any job state when switching to RocksDB. If there's a way to do it then I can worry about RocksDB later.

Thanks!

Josh

[Josh]

Hi Aljoscha,

Thanks, that makes sense. I will start using RocksDB right away then.

Josh

On Wed, Jun 15, 2016 at 1:01 PM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

right now migrating from one state backend to another is not possible. I have it in the back of my head, however, that we should introduce a common serialized representation of state to make this possible in the future. (Both for checkpoints and savepoints, which use the same mechanism underneath.)

Cheers,

Aljoscha

On Wed, 15 Jun 2016 at 00:04 Josh <[hidden email]> wrote:

I'm trying to decide whether to set-up RocksDB now or later.

The state for my Flink jobs right now will be able to fit into memory, so I can use the filesystem state backend. In a few months time it is likely the state will no longer fit into memory, so I will want to use the RocksDB backend.

I was just wondering if it's possible/easy to use savepoints to migrate existing state from the filesystem backend to the RocksDB backend? As I would not want to lose any job state when switching to RocksDB. If there's a way to do it then I can worry about RocksDB later.

Thanks!

Josh