(DEPRECATED) Apache Flink User Mailing List archive.

Could not load the native RocksDB library

Classic

List

Threaded

6 messages Options

Patro, Samya

Could not load the native RocksDB library

Hello,
I am using rocksdb for storing state . But when I run the pipeline I get the error ”Could not load the native RocksDB library” . Kindly can you check the configs and error stacktrace and suggest what am I doing wrong .

Flink version - 1.8.0

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
    <version>1.8.0</version>
</dependency>

This is the flink checkpointing config I have used

executionEnvironment.enableCheckpointing(300000);
executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(50000);

executionEnvironment.getCheckpointConfig().setCheckpointTimeout(600000);
executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

executionEnvironment.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

StateBackend rocksDbBackend = new RocksDBStateBackend(parameter.get("stateBackendPath"),true);
executionEnvironment.setStateBackend(rocksDbBackend);

When I run the pipeline, I get this error

java.lang.Exception: Exception while creating StreamOperatorStateContext.

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)

at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)

at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from any of the 1 provided restore options.

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)

... 5 more

Caused by: java.io.IOException: Could not load the native RocksDB library

at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)

at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)

at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: undefined symbol: malloc_stats_print

at java.lang.ClassLoader$NativeLibrary.load(Native Method)

at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

at java.lang.Runtime.load0(Runtime.java:809)

at java.lang.System.load(System.java:1086)

at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)

... 11 more

Thanks and Regards,
Samya Ranjan Patro
Goldman sachs

Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices

Haibo Sun

Re:Could not load the native RocksDB library

Hi, Samya.Patro

I guess this may be a setup problem. What OS and what version of JDK do you use? You can try upgrading JDK to see if the issue can be solved.

Best,

Haibo

At 2019-07-02 17:16:59, "Patro, Samya" <[hidden email]> wrote:

Hello,
I am using rocksdb for storing state . But when I run the pipeline I get the error ”Could not load the native RocksDB library” . Kindly can you check the configs and error stacktrace and suggest what am I doing wrong .

Flink version - 1.8.0
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
    <version>1.8.0</version>
</dependency>
This is the flink checkpointing config I have used

executionEnvironment.enableCheckpointing(300000);
executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(50000);

executionEnvironment.getCheckpointConfig().setCheckpointTimeout(600000);
executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

executionEnvironment.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

StateBackend rocksDbBackend = new RocksDBStateBackend(parameter.get("stateBackendPath"),true);
executionEnvironment.setStateBackend(rocksDbBackend);

When I run the pipeline, I get this error

java.lang.Exception: Exception while creating StreamOperatorStateContext.

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from any of the 1 provided restore options.

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)

    ... 5 more

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

    ... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: undefined symbol: malloc_stats_print

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)

    ... 11 more

Thanks and Regards,
Samya Ranjan Patro
Goldman sachs
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices

Andrey Zagrebin-3

Re: Could not load the native RocksDB library

Hi Samya,

Additionally to Haibo's answer:

Have you tried the previous 1.7 version of Flink? The Rocksdb version was upgraded in 1.8 version.

Best,

Andrey

On Wed, Jul 3, 2019 at 5:21 AM Haibo Sun <[hidden email]> wrote:

Hi, Samya.Patro

I guess this may be a setup problem. What OS and what version of JDK do you use? You can try upgrading JDK to see if the issue can be solved.

Best,
Haibo

At 2019-07-02 17:16:59, "Patro, Samya" <[hidden email]> wrote:
Hello,
I am using rocksdb for storing state . But when I run the pipeline I get the error ”Could not load the native RocksDB library” . Kindly can you check the configs and error stacktrace and suggest what am I doing wrong .

Flink version - 1.8.0
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
    <version>1.8.0</version>
</dependency>
This is the flink checkpointing config I have used

executionEnvironment.enableCheckpointing(300000);
executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(50000);

executionEnvironment.getCheckpointConfig().setCheckpointTimeout(600000);
executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

executionEnvironment.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

StateBackend rocksDbBackend = new RocksDBStateBackend(parameter.get("stateBackendPath"),true);
executionEnvironment.setStateBackend(rocksDbBackend);

When I run the pipeline, I get this error

java.lang.Exception: Exception while creating StreamOperatorStateContext.

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from any of the 1 provided restore options.

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)

    ... 5 more

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

    ... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: undefined symbol: malloc_stats_print

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)

    ... 11 more

Thanks and Regards,
Samya Ranjan Patro
Goldman sachs
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices

Thad Truman

RE: Could not load the native RocksDB library

Hi Samya,

Were you able to get this resolved? Seeing the same issue here after upgrading to Flink 1.9 from 1.6.

Thanks,

Thad

From: Andrey Zagrebin <[hidden email]>
Sent: Wednesday, July 3, 2019 9:09 AM
To: Haibo Sun <[hidden email]>
Cc: Patro, Samya <[hidden email]>; [hidden email]; Bari, Swapnil <[hidden email]>
Subject: Re: Could not load the native RocksDB library

Hi Samya,

Additionally to Haibo's answer:

Have you tried the previous 1.7 version of Flink? The Rocksdb version was upgraded in 1.8 version.

Best,

Andrey

On Wed, Jul 3, 2019 at 5:21 AM Haibo Sun <[hidden email]> wrote:

Hi, Samya.Patro

I guess this may be a setup problem. What OS and what version of JDK do you use? You can try upgrading JDK to see if the issue can be solved.

Best,

Haibo

At 2019-07-02 17:16:59, "Patro, Samya" <[hidden email]> wrote:
Hello,
I am using rocksdb for storing state . But when I run the pipeline I get the error ”Could not load the native RocksDB library” . Kindly can you check the configs and error stacktrace and suggest what am I doing wrong .

Flink version - 1.8.0
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
    <version>1.8.0</version>
</dependency>
This is the flink checkpointing config I have used

executionEnvironment.enableCheckpointing(300000);
executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(50000);

executionEnvironment.getCheckpointConfig().setCheckpointTimeout(600000);
executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

executionEnvironment.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

StateBackend rocksDbBackend = new RocksDBStateBackend(parameter.get("stateBackendPath"),true);
executionEnvironment.setStateBackend(rocksDbBackend);

When I run the pipeline, I get this error

java.lang.Exception: Exception while creating StreamOperatorStateContext.

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from any of the 1 provided restore options.

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)

    ... 5 more

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

    ... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: undefined symbol: malloc_stats_print

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)

    ... 11 more

Thanks and Regards,
Samya Ranjan Patro
Goldman sachs
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices

Congxian Qiu

Re: Could not load the native RocksDB library

FYI

Maybe this is an env problem. I encountered this problem when running flink 1.9 on k8s, but it was success when running on yarn. did not figure out why this happened, will update here after find it out.

Best,

Congxian

Thad Truman <[hidden email]> 于2019年10月23日周三上午1:33写道：

Hi Samya,

Were you able to get this resolved? Seeing the same issue here after upgrading to Flink 1.9 from 1.6.

Thanks,

Thad

From: Andrey Zagrebin <[hidden email]>
Sent: Wednesday, July 3, 2019 9:09 AM
To: Haibo Sun <[hidden email]>
Cc: Patro, Samya <[hidden email]>; [hidden email]; Bari, Swapnil <[hidden email]>
Subject: Re: Could not load the native RocksDB library
Hi Samya,

Additionally to Haibo's answer:

Have you tried the previous 1.7 version of Flink? The Rocksdb version was upgraded in 1.8 version.

Best,

Andrey
On Wed, Jul 3, 2019 at 5:21 AM Haibo Sun <[hidden email]> wrote:
Hi, Samya.Patro

I guess this may be a setup problem. What OS and what version of JDK do you use? You can try upgrading JDK to see if the issue can be solved.

Best,

Haibo

At 2019-07-02 17:16:59, "Patro, Samya" <[hidden email]> wrote:
Hello,
I am using rocksdb for storing state . But when I run the pipeline I get the error ”Could not load the native RocksDB library” . Kindly can you check the configs and error stacktrace and suggest what am I doing wrong .

Flink version - 1.8.0
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
    <version>1.8.0</version>
</dependency>
This is the flink checkpointing config I have used

executionEnvironment.enableCheckpointing(300000);
executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(50000);

executionEnvironment.getCheckpointConfig().setCheckpointTimeout(600000);
executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

executionEnvironment.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

StateBackend rocksDbBackend = new RocksDBStateBackend(parameter.get("stateBackendPath"),true);
executionEnvironment.setStateBackend(rocksDbBackend);

When I run the pipeline, I get this error

java.lang.Exception: Exception while creating StreamOperatorStateContext.

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from any of the 1 provided restore options.

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)

    ... 5 more

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

    ... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: undefined symbol: malloc_stats_print

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)

    ... 11 more

Thanks and Regards,
Samya Ranjan Patro
Goldman sachs
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices

Patro, Samya

RE: Could not load the native RocksDB library

In reply to this post by Thad Truman

Hello Thad,
In my case , the issue was fixed after upgrading the os version , and gcc version.

From: Thad Truman [mailto:[hidden email]]
Sent: Tuesday, October 22, 2019 11:03 PM
To: Andrey Zagrebin; Haibo Sun
Cc: Patro, Samya [Engineering]; [hidden email]; Bari, Swapnil [Engineering]
Subject: RE: Could not load the native RocksDB library

Hi Samya,

Were you able to get this resolved? Seeing the same issue here after upgrading to Flink 1.9 from 1.6.

Thanks,

Thad

Hi Samya,

Additionally to Haibo's answer:

Have you tried the previous 1.7 version of Flink? The Rocksdb version was upgraded in 1.8 version.

Best,

Andrey

On Wed, Jul 3, 2019 at 5:21 AM Haibo Sun <[hidden email]> wrote:

Hi, Samya.Patro

I guess this may be a setup problem. What OS and what version of JDK do you use? You can try upgrading JDK to see if the issue can be solved.

Best,

Haibo

At 2019-07-02 17:16:59, "Patro, Samya" <[hidden email]> wrote:
Hello,
I am using rocksdb for storing state . But when I run the pipeline I get the error ”Could not load the native RocksDB library” . Kindly can you check the configs and error stacktrace and suggest what am I doing wrong .

Flink version - 1.8.0
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
    <version>1.8.0</version>
</dependency>
This is the flink checkpointing config I have used

executionEnvironment.enableCheckpointing(300000);
executionEnvironment.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE);
executionEnvironment.getCheckpointConfig().setMinPauseBetweenCheckpoints(50000);

executionEnvironment.getCheckpointConfig().setCheckpointTimeout(600000);
executionEnvironment.getCheckpointConfig().setMaxConcurrentCheckpoints(1);

executionEnvironment.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);

StateBackend rocksDbBackend = new RocksDBStateBackend(parameter.get("stateBackendPath"),true);
executionEnvironment.setStateBackend(rocksDbBackend);

When I run the pipeline, I get this error

java.lang.Exception: Exception while creating StreamOperatorStateContext.

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:195)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:250)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for StreamFlatMap_9dd63673dd41ea021b896d5203f3ba7c_(1/5) from any of the 1 provided restore options.

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)

    ... 5 more

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:911)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:482)

    at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)

    at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)

    ... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ee961b2f013f7d5baabbc1cb2b0b87d7/librocksdbjni-linux64.so: undefined symbol: malloc_stats_print

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:888)

    ... 11 more

Thanks and Regards,
Samya Ranjan Patro
Goldman sachs
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices