RocksDBMapState example?

[NEKRASSOV, ALEXEI]

Hi,

 

I’d like to use RocksDB to store a key-value mapping table (with 45 million keys).

Can someone please point me to an example of RocksDBMapState() constructor invocation? Or an explanation of constructor arguments?..

 

Thanks,

Alex Nekrassov

[hidden email]

[Ted Yu]

Hi,

Have you looked at the ctor call in :

flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

around line 1261 ?

Cheers

[NEKRASSOV, ALEXEI]

I looked at that code, but I’m still not clear.

 

new RocksDBMapState<>(columnFamily, namespaceSerializer, stateDesc, this);

 

columnFamily is determined by 50-line function; is this necessary for a simple use case like mine? What should I use as state descriptor in that function?..

Last argument is set to “this”; does this mean I need to implement AbstractKeyedStateBackend, like RocksDBKeyedStateBackend does?

 

Again, I’m looking for a simple equivalent to

new HashMap();

 

or to

JedisPool pool = new JedisPool(new JedisPoolConfig(), redisHost);

Jedis jedis = pool.getResource();

 

Thanks,

Alex

 

From: Ted Yu [mailto:[hidden email]]
Sent: Monday, April 09, 2018 11:48 AM
To: user <[hidden email]>
Subject: Re: RocksDBMapState example?

 

Hi,

Have you looked at the ctor call in :

flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java

 

around line 1261 ?

 

Cheers

[Dawid Wysakowicz]

Hi Alexei,

You should not use RocksDBMapState directly. Have you went through the doc page regarding working with state[1]?
I think you want to use KeyedState, assuming the size of your keyspace. Probably a way to go would be to key your stream and then even ValueState (which will be scoped to that key) might be sufficient.
You can configure flink further to use RocksDB as the underlying state backend[2]

Regards,
Dawid

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state
[2] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#state-backends

> On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI <[hidden email]> wrote:
>
> Hi,
>
> I’d like to use RocksDB to store a key-value mapping table (with 45 million keys).
> Can someone please point me to an example of RocksDBMapState() constructor invocation? Or an explanation of constructor arguments?..
>
> Thanks,
> Alex Nekrassov
> [hidden email]

[NEKRASSOV, ALEXEI]

Yes, I've read the documentation on working with state.
It talks about MapState<UK, UV>. When I looked at Javadoc, I learned that MapState is an interface, with RocksDBMapState as one of the implementing classes.

I'm not sure what you mean by KeyedState; I don't see a class with that name.

I'm not clear how ValueState can be used to store key-value mapping. Can you please clarify?

Thanks,
Alex

-----Original Message-----
From: Dawid Wysakowicz [mailto:[hidden email]]
Sent: Tuesday, April 10, 2018 8:54 AM
To: NEKRASSOV, ALEXEI <[hidden email]>
Cc: [hidden email]
Subject: Re: RocksDBMapState example?

Hi Alexei,

You should not use RocksDBMapState directly. Have you went through the doc page regarding working with state[1]?
I think you want to use KeyedState, assuming the size of your keyspace. Probably a way to go would be to key your stream and then even ValueState (which will be scoped to that key) might be sufficient.
You can configure flink further to use RocksDB as the underlying state backend[2]

Regards,
Dawid

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state
[2] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#state-backends

> On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI <[hidden email]> wrote:
>
> Hi,
>
> I’d like to use RocksDB to store a key-value mapping table (with 45 million keys).
> Can someone please point me to an example of RocksDBMapState() constructor invocation? Or an explanation of constructor arguments?..
>
> Thanks,
> Alex Nekrassov
> [hidden email]

[Ted Yu]

For KeyedState, apart from https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#keyed-state-and-operator-state , you can refer to docs/dev/migration.md :

    public void initializeState(FunctionInitializationContext context) throws Exception {

        counter = context.getKeyedStateStore().getState(

            new ValueStateDescriptor<>("counter", Integer.class, 0));

FYI

On Tue, Apr 10, 2018 at 7:24 AM, NEKRASSOV, ALEXEI <[hidden email]> wrote:

Yes, I've read the documentation on working with state.
It talks about MapState<UK, UV>. When I looked at Javadoc, I learned that MapState is an interface, with RocksDBMapState as one of the implementing classes.

I'm not sure what you mean by KeyedState; I don't see a class with that name.

I'm not clear how ValueState can be used to store key-value mapping. Can you please clarify?

Thanks,
Alex

-----Original Message-----
From: Dawid Wysakowicz [mailto:[hidden email]]
Sent: Tuesday, April 10, 2018 8:54 AM
To: NEKRASSOV, ALEXEI <[hidden email]>
Cc: [hidden email]
Subject: Re: RocksDBMapState example?

Hi Alexei,

You should not use RocksDBMapState directly. Have you went through the doc page regarding working with state[1]?
I think you want to use KeyedState, assuming the size of your keyspace. Probably a way to go would be to key your stream and then even ValueState (which will be scoped to that key) might be sufficient.
You can configure flink further to use RocksDB as the underlying state backend[2]

Regards,
Dawid

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html#working-with-state
[2] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html#state-backends

> On 9 Apr 2018, at 17:41, NEKRASSOV, ALEXEI <[hidden email]> wrote:
>
> Hi,
>
> I’d like to use RocksDB to store a key-value mapping table (with 45 million keys).
> Can someone please point me to an example of RocksDBMapState() constructor invocation? Or an explanation of constructor arguments?..
>
> Thanks,
> Alex Nekrassov
> [hidden email]