Keyed raw state - example

[bastien dine]

Hello everyone,

I would like to know if anybody has a working example on how to declare a keyed raw state ( in my case a keyedprocessoperator) and how to use  it in my UDF (keyedprocessfunction)?

Basicaly we have a huge problem with a ValueState w Rocksdb, getting serialized for every element ( need to access it and update) so it's taking a crazy amount of time and we would like to have it serialized only on snapshot, so using Raw state is a possible good solution, 

But i cannot find anyexample of it :/

Thanks and best regards,

[Congxian Qiu]

Hi

   Currently, I think you can ref the implementation of timerservice[1] which used the raw keyed state, the snapshot happens in AbstractStreamOperator#snapshotState(), for using Raw State you need to implement a new operator[2]. There is an issue wants to give some example for raw state[2]

[1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManager.java

[2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#raw-and-managed-state

[3] https://issues.apache.org/jira/browse/FLINK-14379

bastien dine <[hidden email]> 于2019年11月16日周六 上午5:57写道：

Hello everyone,

I would like to know if anybody has a working example on how to declare a keyed raw state ( in my case a keyedprocessoperator) and how to use  it in my UDF (keyedprocessfunction)?

Basicaly we have a huge problem with a ValueState w Rocksdb, getting serialized for every element ( need to access it and update) so it's taking a crazy amount of time and we would like to have it serialized only on snapshot, so using Raw state is a possible good solution, 

But i cannot find anyexample of it :/

Thanks and best regards,

Bastien DINE
Freelance
Data Architect / Software Engineer / Sysadmin
http://bastiendine.io

   

[bastien dine]

Hello Congxian,

Thanks for your response,

Don't you have an example with an Operator extending the AbstractUdfStreamOperator?

Using the context.getRawKeyedStateInputs() (& output to snapshots)

TimeService is reimplementing the whole stuff :/

Le lun. 18 nov. 2019 à 03:19, Congxian Qiu <[hidden email]> a écrit :

Hi

   Currently, I think you can ref the implementation of timerservice[1] which used the raw keyed state, the snapshot happens in AbstractStreamOperator#snapshotState(), for using Raw State you need to implement a new operator[2]. There is an issue wants to give some example for raw state[2]

[1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManager.java

[2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#raw-and-managed-state

[3] https://issues.apache.org/jira/browse/FLINK-14379

Best,

Congxian

bastien dine <[hidden email]> 于2019年11月16日周六 上午5:57写道：

Hello everyone,

I would like to know if anybody has a working example on how to declare a keyed raw state ( in my case a keyedprocessoperator) and how to use  it in my UDF (keyedprocessfunction)?

Basicaly we have a huge problem with a ValueState w Rocksdb, getting serialized for every element ( need to access it and update) so it's taking a crazy amount of time and we would like to have it serialized only on snapshot, so using Raw state is a possible good solution, 

But i cannot find anyexample of it :/

Thanks and best regards,

Bastien DINE
Freelance
Data Architect / Software Engineer / Sysadmin
http://bastiendine.io