Flink 1.3.2 RocksDB map segment fail if configured as state backend

[Andrea Spina]

Hi everybody,

I run with a Flink 1.3.2 installation on a Red Hat Enterprise Linux Server and I'm not able to set rocksdb as state.backend due to this error whenever I try to deploy any job:

java.lang.IllegalStateException: Could not initialize keyed state backend.

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:678)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:666)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:252)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)

    at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:560)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:298)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:756)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)

    ... 6 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: failed to map segment from shared object: Operation not permitted

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:537)

    ... 9 more

Machine detail

NAME="Red Hat Enterprise Linux Server"

VERSION="7.5 (Maipo)"

ID="rhel"

ID_LIKE="fedora"

Cpu architecture (cat /proc/cpuinfo)

processor    : 0

vendor_id    : GenuineIntel

cpu family    : 6

model        : 45

model name    : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

stepping    : 2

microcode    : 0x710

cpu MHz        : 2200.000

cache size    : 20480 KB

physical id    : 0

siblings    : 4

core id        : 0

cpu cores    : 4

apicid        : 0

initial apicid    : 0

fpu        : yes

fpu_exception    : yes

cpuid level    : 13

wp        : yes

flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm epb dtherm ida arat pln pts

bogomips    : 4400.00

clflush size    : 64

cache_alignment    : 64

address sizes    : 40 bits physical, 48 bits virtual

I found similar errors related to different systems [1] and similar problems with rocksdb related to endianness [2], but mine sounds different.

Since the upgrade to newer Flink version atm might be painful, are there any reason behind this exception, and is a workaround existing?

Thank you so much,

Andrea

[1] - https://communities.ca.com/thread/241773398-em-failed-to-start-tmplibrocksdbjni8883977260053861907so-failed-to-map-segment-from-shared-object-operation-not-permitted

[2] - http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Unable-to-use-Flink-RocksDB-state-backend-due-to-endianness-mismatch-td13473.html

--

[Kostas Kloudas]

Hi Andrea,

I think that Andrey and Stefan (cc’ed) may be able to help.

Kostas

On Sep 17, 2018, at 11:37 AM, Andrea Spina <[hidden email]> wrote:

Hi everybody,

I run with a Flink 1.3.2 installation on a Red Hat Enterprise Linux Server and I'm not able to set rocksdb as state.backend due to this error whenever I try to deploy any job:

java.lang.IllegalStateException: Could not initialize keyed state backend.

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:678)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:666)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:252)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)

    at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:560)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:298)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:756)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)

    ... 6 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: failed to map segment from shared object: Operation not permitted

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:537)

    ... 9 more

Machine detail

NAME="Red Hat Enterprise Linux Server"

VERSION="7.5 (Maipo)"

ID="rhel"

ID_LIKE="fedora"

Cpu architecture (cat /proc/cpuinfo)

processor    : 0

vendor_id    : GenuineIntel

cpu family    : 6

model        : 45

model name    : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

stepping    : 2

microcode    : 0x710

cpu MHz        : 2200.000

cache size    : 20480 KB

physical id    : 0

siblings    : 4

core id        : 0

cpu cores    : 4

apicid        : 0

initial apicid    : 0

fpu        : yes

fpu_exception    : yes

cpuid level    : 13

wp        : yes

flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm epb dtherm ida arat pln pts

bogomips    : 4400.00

clflush size    : 64

cache_alignment    : 64

address sizes    : 40 bits physical, 48 bits virtual

I found similar errors related to different systems [1] and similar problems with rocksdb related to endianness [2], but mine sounds different.

Since the upgrade to newer Flink version atm might be painful, are there any reason behind this exception, and is a workaround existing?

Thank you so much,

Andrea

[1] - https://communities.ca.com/thread/241773398-em-failed-to-start-tmplibrocksdbjni8883977260053861907so-failed-to-map-segment-from-shared-object-operation-not-permitted

[2] - http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Unable-to-use-Flink-RocksDB-state-backend-due-to-endianness-mismatch-td13473.html

--

[Stefan Richter]

Hi,

I think the exception pretty much says what is wrong, the native library cannot be mapped into the process because of some access rights problem. Please make sure that your path /tmp has the exec right.

Best,

Stefan

Am 17.09.2018 um 11:37 schrieb Andrea Spina <[hidden email]>:

Hi everybody,

I run with a Flink 1.3.2 installation on a Red Hat Enterprise Linux Server and I'm not able to set rocksdb as state.backend due to this error whenever I try to deploy any job:

java.lang.IllegalStateException: Could not initialize keyed state backend.

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:678)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:666)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:252)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)

    at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:560)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:298)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:756)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)

    ... 6 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: failed to map segment from shared object: Operation not permitted

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:537)

    ... 9 more

Machine detail

NAME="Red Hat Enterprise Linux Server"

VERSION="7.5 (Maipo)"

ID="rhel"

ID_LIKE="fedora"

Cpu architecture (cat /proc/cpuinfo)

processor    : 0

vendor_id    : GenuineIntel

cpu family    : 6

model        : 45

model name    : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

stepping    : 2

microcode    : 0x710

cpu MHz        : 2200.000

cache size    : 20480 KB

physical id    : 0

siblings    : 4

core id        : 0

cpu cores    : 4

apicid        : 0

initial apicid    : 0

fpu        : yes

fpu_exception    : yes

cpuid level    : 13

wp        : yes

flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm epb dtherm ida arat pln pts

bogomips    : 4400.00

clflush size    : 64

cache_alignment    : 64

address sizes    : 40 bits physical, 48 bits virtual

I found similar errors related to different systems [1] and similar problems with rocksdb related to endianness [2], but mine sounds different.

Since the upgrade to newer Flink version atm might be painful, are there any reason behind this exception, and is a workaround existing?

Thank you so much,

Andrea

[1] - https://communities.ca.com/thread/241773398-em-failed-to-start-tmplibrocksdbjni8883977260053861907so-failed-to-map-segment-from-shared-object-operation-not-permitted

[2] - http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Unable-to-use-Flink-RocksDB-state-backend-due-to-endianness-mismatch-td13473.html

--

[Andrea Spina]

Hi Stefan, thank you. You were right. It was an unexpected permits issue. Thank you again.

2018-09-17 16:50 GMT+02:00 Stefan Richter <[hidden email]>:

Hi,

I think the exception pretty much says what is wrong, the native library cannot be mapped into the process because of some access rights problem. Please make sure that your path /tmp has the exec right.

Best,

Stefan

Am 17.09.2018 um 11:37 schrieb Andrea Spina <[hidden email]>:

Hi everybody,

I run with a Flink 1.3.2 installation on a Red Hat Enterprise Linux Server and I'm not able to set rocksdb as state.backend due to this error whenever I try to deploy any job:

java.lang.IllegalStateException: Could not initialize keyed state backend.

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:321)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:217)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:678)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:666)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:252)

    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)

    at java.lang.Thread.run(Thread.java:748)

Caused by: java.io.IOException: Could not load the native RocksDB library

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:560)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:298)

    at org.apache.flink.streaming.runtime.tasks.StreamTask.createKeyedStateBackend(StreamTask.java:756)

    at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initKeyedState(AbstractStreamOperator.java:311)

    ... 6 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: /tmp/rocksdb-lib-ab7e3d3688fe883981ec37668bf2cbc3/librocksdbjni-linux64.so: failed to map segment from shared object: Operation not permitted

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.ensureRocksDBIsLoaded(RocksDBStateBackend.java:537)

    ... 9 more

Machine detail

NAME="Red Hat Enterprise Linux Server"

VERSION="7.5 (Maipo)"

ID="rhel"

ID_LIKE="fedora"

Cpu architecture (cat /proc/cpuinfo)

processor    : 0

vendor_id    : GenuineIntel

cpu family    : 6

model        : 45

model name    : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

stepping    : 2

microcode    : 0x710

cpu MHz        : 2200.000

cache size    : 20480 KB

physical id    : 0

siblings    : 4

core id        : 0

cpu cores    : 4

apicid        : 0

initial apicid    : 0

fpu        : yes

fpu_exception    : yes

cpuid level    : 13

wp        : yes

flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm epb dtherm ida arat pln pts

bogomips    : 4400.00

clflush size    : 64

cache_alignment    : 64

address sizes    : 40 bits physical, 48 bits virtual

I found similar errors related to different systems [1] and similar problems with rocksdb related to endianness [2], but mine sounds different.

Since the upgrade to newer Flink version atm might be painful, are there any reason behind this exception, and is a workaround existing?

Thank you so much,

Andrea

[1] - https://communities.ca.com/thread/241773398-em-failed-to-start-tmplibrocksdbjni8883977260053861907so-failed-to-map-segment-from-shared-object-operation-not-permitted

[2] - http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Unable-to-use-Flink-RocksDB-state-backend-due-to-endianness-mismatch-td13473.html

--

--