Compression with rocksdb backed state

[deepthi Sridharan]

I am trying to understand this section on compression of checkpoints which has me a bit confused https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression

Could you please confirm if my understanding is correct: 

1. Compression is disabled by default for full snapshots and needs to be turned on if required. 

2. Current versions do not support changing the compression type (snappy by default) when enabled. Future versions will support configuring the compression type.

3. Compression is enabled by default for incremental snapshot.

4. The compression type cannot be configured for incremental savepoints.

--

[Timo Walther]

Hi Deepthi,

1. Correct
2. Correct
3. Incremental snapshots simply manage references to RocksDB's sstables.
You can find a full explanation here [1]. Thus, the payload is a
blackbox for Flink and Flink's compression flag has no impact. So we
fully rely what RocksDB offers.
4. Correct

I hope this helps.

Regards,
Timo


[1]
https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html


On 07.04.21 22:04, deepthi Sridharan wrote:

> I am trying to understand this section on compression of checkpoints
> which has me a bit confused
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression 
> <https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression>
>
> Could you please confirm if my understanding is correct:
> 1. Compression is disabled by default for full snapshots and needs to be
> turned on if required.
> 2. Current versions do not support changing the compression type (snappy
> by default) when enabled. Future versions will support configuring the
> compression type.
> 3. Compression is enabled by default for incremental snapshot.
> 4. The compression type cannot be configured for incremental savepoints.
>
> --
> Regards,
> Deepthi

[deepthi Sridharan]

Thank you, that makes sense.

On Thu, Apr 8, 2021 at 12:37 AM Timo Walther <[hidden email]> wrote:

Hi Deepthi,

1. Correct
2. Correct
3. Incremental snapshots simply manage references to RocksDB's sstables.
You can find a full explanation here [1]. Thus, the payload is a
blackbox for Flink and Flink's compression flag has no impact. So we
fully rely what RocksDB offers.
4. Correct

I hope this helps.

Regards,
Timo


[1]
https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html


On 07.04.21 22:04, deepthi Sridharan wrote:
> I am trying to understand this section on compression of checkpoints
> which has me a bit confused
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression
> <https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression>
>
> Could you please confirm if my understanding is correct:
> 1. Compression is disabled by default for full snapshots and needs to be
> turned on if required.
> 2. Current versions do not support changing the compression type (snappy
> by default) when enabled. Future versions will support configuring the
> compression type.
> 3. Compression is enabled by default for incremental snapshot.
> 4. The compression type cannot be configured for incremental savepoints.
>
> --
> Regards,
> Deepthi

--