Re: Rocksdb Serialization issue

Posted by aldu29 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Rocksdb-Serialization-issue-tp33388p33413.html

Hello Arvid,

After some investigations with the help of my colleague we finally found the root cause.
In order to improve the init of the state, I've created some threads to parallelize the read of bucket files.
This is a temporary solution because I've planned to use the State Processor API.
Here after an abstract of the code:

ExecutorService executorService = Executors.newFixedThreadPool(initStateMaxThreads);
for (FileStatus bucketFile : xxx) {
    executorService.submit(
            () -> {
        try {
            readBucketFct(XXX); // Update the state with the bucket content...
        } catch (Exception e) {
           ....
        }
    });
}
executorService.shutdown();
boolean terminated = executorService.awaitTermination(initStateTimeoutSeconds, TimeUnit.SECONDS);
if ((!terminated) || (readMetaErrors.get() > 0)) {
    throw new SinkException("Init state failed...") ;
}

After some tests: if I use one thead in my executorService it works. But with 2 threads the job fails.
Can I mitigate this behaviour (in waiting the switch to the State Processor API) ?

Thanks

David

Le jeu. 5 mars 2020 à 08:06, Arvid Heise <[hidden email]> a écrit :

Hi David,

the obvious reason is that your state stored an enum value that is not present anymore. It tries to deserialize the 512. entry in your enum that is not available.

However, since it's highly unlikely that you actually have that many enum values in the same enum class, we are actually looking at a corrupt stream, which is hard to fix. Could you describe which state you have?

Did you upgrade Flink or your application? If it's Flink, it's a bug. If it's application, it may be that state is incompatible and would need to be migrated.

Did you restart from checkpoint or savepoint?

On Thu, Mar 5, 2020 at 1:14 AM David Morin <[hidden email]> wrote:

Hello,

I have this Exception in my datastream app and I can't find the root cause.
I consume data from Kafka and it fails when I try to get a value from my MapState in RocksDB.
It was working in previous release of my app but I can't find the cause of this error.

java.lang.ArrayIndexOutOfBoundsException: 512
        at org.apache.flink.api.common.typeutils.base.EnumSerializer.deserialize(EnumSerializer.java:130)
        at org.apache.flink.api.common.typeutils.base.EnumSerializer.deserialize(EnumSerializer.java:50)
        at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:143)
        at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize(TupleSerializer.java:37)
        at org.apache.flink.contrib.streaming.state.RocksDBMapState.deserializeUserValue(RocksDBMapState.java:344)
        at org.apache.flink.contrib.streaming.state.RocksDBMapState.get(RocksDBMapState.java:123)
        at org.apache.flink.runtime.state.UserFacingMapState.get(UserFacingMapState.java:47)
..

Flink version: 1.9.2