Prefix Seek RocksDB

[Rex Fenley]

Hello!

I'm wondering if Flink RocksDB state backend is pre-configured to have Prefix Seeks enabled, such as for Joins and Aggs on the TableAPI [1]? If not, what's the easiest way to configure this? I'd imagine this would be beneficial.

Thanks!

[1] https://github.com/facebook/rocksdb/wiki/Prefix-Seek

--

[Yun Tang]

Hi Rex, 

You could configure prefix seek via RocksDB's column family options [1]. Be careful to use correct prefix extractor.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#passing-options-factory-to-rocksdb

Best

 

---

From: Rex Fenley <[hidden email]>
Sent: Tuesday, March 16, 2021 8:29
To: user <[hidden email]>
Cc: Brad Davis <[hidden email]>
Subject: Prefix Seek RocksDB

 

Hello!

I'm wondering if Flink RocksDB state backend is pre-configured to have Prefix Seeks enabled, such as for Joins and Aggs on the TableAPI [1]? If not, what's the easiest way to configure this? I'd imagine this would be beneficial.

Thanks!

[1] https://github.com/facebook/rocksdb/wiki/Prefix-Seek

--

[Rex Fenley]

Thanks for the input, I'll look more into that.

Does your answer then imply that Joins and Aggs do not inherently always use prefix seeks? I'd imagine that the join key on join and groupby key on aggs would always be used as prefix keys. Is this not the case?

Also, is there good information on what the correct prefix extractor is for Flink? This feature is something I only just discovered so I was hoping to gain clarity.

Thanks

On Mon, Mar 15, 2021 at 8:33 PM Yun Tang <[hidden email]> wrote:

Hi Rex, 

You could configure prefix seek via RocksDB's column family options [1]. Be careful to use correct prefix extractor.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#passing-options-factory-to-rocksdb

Best

 

---

From: Rex Fenley <[hidden email]>
Sent: Tuesday, March 16, 2021 8:29
To: user <[hidden email]>
Cc: Brad Davis <[hidden email]>
Subject: Prefix Seek RocksDB

 

Hello!

I'm wondering if Flink RocksDB state backend is pre-configured to have Prefix Seeks enabled, such as for Joins and Aggs on the TableAPI [1]? If not, what's the easiest way to configure this? I'd imagine this would be beneficial.

Thanks!

[1] https://github.com/facebook/rocksdb/wiki/Prefix-Seek

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--

[Yun Tang]

Hi Rex,

Prefix seek iterator has not ever been used in Flink when seeking. I hope you could first read more details about this from RocksDB wiki as prefix extractor could impact the performance.

Best

Yun Tang

---

From: Rex Fenley <[hidden email]>
Sent: Wednesday, March 17, 2021 2:02
To: Yun Tang <[hidden email]>
Cc: user <[hidden email]>; Brad Davis <[hidden email]>
Subject: Re: Prefix Seek RocksDB

 

Thanks for the input, I'll look more into that.

Does your answer then imply that Joins and Aggs do not inherently always use prefix seeks? I'd imagine that the join key on join and groupby key on aggs would always be used as prefix keys. Is this not the case?

Also, is there good information on what the correct prefix extractor is for Flink? This feature is something I only just discovered so I was hoping to gain clarity.

Thanks

On Mon, Mar 15, 2021 at 8:33 PM Yun Tang <[hidden email]> wrote:

Hi Rex, 

You could configure prefix seek via RocksDB's column family options [1]. Be careful to use correct prefix extractor.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backends.html#passing-options-factory-to-rocksdb

Best

 

---

From: Rex Fenley <[hidden email]>
Sent: Tuesday, March 16, 2021 8:29
To: user <[hidden email]>
Cc: Brad Davis <[hidden email]>
Subject: Prefix Seek RocksDB

 

Hello!

I'm wondering if Flink RocksDB state backend is pre-configured to have Prefix Seeks enabled, such as for Joins and Aggs on the TableAPI [1]? If not, what's the easiest way to configure this? I'd imagine this would be beneficial.

Thanks!

[1] https://github.com/facebook/rocksdb/wiki/Prefix-Seek

--

Rex Fenley  |  Software Engineer - Mobile and Backend

Remind.com |  BLOG  |  FOLLOW US  |  LIKE US

--