Error "Failed to load native Mesos library from" when I run Flink on a compiled version of Apache Mesos

[Felipe Gutierrez]

Hi,

I am compiling mesos from source code and trying to execute Flink on it. For some reason I am having the error that "Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib". I searched on internet and everybody is saying to load the library paths like I did below. But the error still remains.

I am able to deploy a dummy task (infinit loop) on mesos and it is running all right.

flink@cow-11:~/flink-1.9.0$ export MESOS_NATIVE_JAVA_LIBRARY="/home/felipe/workspace-vsc/mesos/build/src/.libs/libmesos.so"

flink@cow-11:~/flink-1.9.0$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/felipe/workspace-vsc/mesos/build/src/.libs/
flink@cow-11:~/flink-1.9.0$ PATH=$PATH:/home/felipe/workspace-vsc/mesos/build/bin/

flink-1.9.0$ sudo ./bin/mesos-appmaster.sh
Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib

Thanks,

Felipe

[Rui Li]

Hey Felipe,

I haven't tried to run Flink on Mesos, but I guess you can try exporting those ENV variables so that Flink can see them. Alternatively you can try something like

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/felipe/workspace-vsc/mesos/build/src/.libs/ ./bin/mesos-appmaster.sh

If the problem persists, please provide detailed stack traces/error messages which should be found in log files.

On Wed, Sep 18, 2019 at 12:34 AM Felipe Gutierrez <[hidden email]> wrote:

Hi,

I am compiling mesos from source code and trying to execute Flink on it. For some reason I am having the error that "Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib". I searched on internet and everybody is saying to load the library paths like I did below. But the error still remains.

I am able to deploy a dummy task (infinit loop) on mesos and it is running all right.

flink@cow-11:~/flink-1.9.0$ export MESOS_NATIVE_JAVA_LIBRARY="/home/felipe/workspace-vsc/mesos/build/src/.libs/libmesos.so"

flink@cow-11:~/flink-1.9.0$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/felipe/workspace-vsc/mesos/build/src/.libs/
flink@cow-11:~/flink-1.9.0$ PATH=$PATH:/home/felipe/workspace-vsc/mesos/build/bin/

flink-1.9.0$ sudo ./bin/mesos-appmaster.sh
Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib

Thanks,

Felipe

--

[Felipe Gutierrez]

thanks Rui,

the problem still persists after I try to start Flink on Mesos with your command.

This is what I have on conf/flink-conf.yaml

mesos.master: 127.0.0.1:5050
mesos.initial-tasks: 1
mesos.resourcemanager.tasks.container.type: mesos
jobmanager.heap.mb: 1024
jobmanager.web.address: 127.0.0.1
jobmanager.web.port: 8081
mesos.resourcemanager.tasks.cpus: 2.0

This is the variable that I export and the command that you suggested.

export MESOS_NATIVE_JAVA_LIBRARY="/home/felipe/workspace-vsc/mesos/build/src/.libs/libmesos.so"
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/felipe/workspace-vsc/mesos/build/src/.libs/
PATH=$PATH:/home/felipe/workspace-vsc/mesos/build/bin/

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/felipe/workspace-vsc/mesos/build/src/.libs/ /home/flink/flink-1.9.0/bin/mesos-appmaster.sh

Here is the stack trace error:

2019-09-19 11:36:24,861 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  - Registering as new framework.
2019-09-19 11:36:24,861 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  - --------------------------------------------------------------------------------
2019-09-19 11:36:24,861 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -  Mesos Info:
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Master URL: 127.0.0.1:5050
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -  Framework Info:
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     ID: (none)
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Name: Flink
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Failover Timeout (secs): 604800.0
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Role: *
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Capabilities: (none)
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Principal: (none)
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Host: 127.0.0.1
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  -     Web UI: http://127.0.0.1:8081
2019-09-19 11:36:24,862 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  - --------------------------------------------------------------------------------
2019-09-19 11:36:24,938 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.tcp://flink@127.0.0.1:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000
2019-09-19 11:36:24,942 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.
2019-09-19 11:36:24,947 INFO  org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  - ResourceManager akka.tcp://flink@127.0.0.1:6123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
2019-09-19 11:36:24,951 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl  - Starting the SlotManager.
2019-09-19 11:36:26,716 ERROR org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager  - Fatal error occurred in ResourceManager.
java.lang.UnsatisfiedLinkError: org.apache.mesos.MesosSchedulerDriver.initialize()V
at org.apache.mesos.MesosSchedulerDriver.initialize(Native Method)
at org.apache.mesos.MesosSchedulerDriver.<init>(MesosSchedulerDriver.java:192)
at org.apache.flink.mesos.util.MesosConfiguration.createDriver(MesosConfiguration.java:117)
at org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager.prepareLeadershipAsync(MesosResourceManager.java:286)
at org.apache.flink.runtime.resourcemanager.ResourceManager.tryAcceptLeadership(ResourceManager.java:930)
at org.apache.flink.runtime.resourcemanager.ResourceManager.lambda$grantLeadership$9(ResourceManager.java:896)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:88)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-09-19 11:36:26,719 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Fatal error occurred in the cluster entrypoint.
java.lang.UnsatisfiedLinkError: org.apache.mesos.MesosSchedulerDriver.initialize()V
at org.apache.mesos.MesosSchedulerDriver.initialize(Native Method)
at org.apache.mesos.MesosSchedulerDriver.<init>(MesosSchedulerDriver.java:192)
at org.apache.flink.mesos.util.MesosConfiguration.createDriver(MesosConfiguration.java:117)
at org.apache.flink.mesos.runtime.clusterframework.MesosResourceManager.prepareLeadershipAsync(MesosResourceManager.java:286)
at org.apache.flink.runtime.resourcemanager.ResourceManager.tryAcceptLeadership(ResourceManager.java:930)
at org.apache.flink.runtime.resourcemanager.ResourceManager.lambda$grantLeadership$9(ResourceManager.java:896)
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:88)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-09-19 11:36:26,728 INFO  org.apache.flink.runtime.blob.BlobServer                      - Stopped BLOB server at 0.0.0.0:37375

On Wed, Sep 18, 2019 at 4:53 AM Rui Li <[hidden email]> wrote:

Hey Felipe,

I haven't tried to run Flink on Mesos, but I guess you can try exporting those ENV variables so that Flink can see them. Alternatively you can try something like

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/felipe/workspace-vsc/mesos/build/src/.libs/ ./bin/mesos-appmaster.sh

If the problem persists, please provide detailed stack traces/error messages which should be found in log files.

On Wed, Sep 18, 2019 at 12:34 AM Felipe Gutierrez <[hidden email]> wrote:

Hi,

I am compiling mesos from source code and trying to execute Flink on it. For some reason I am having the error that "Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib". I searched on internet and everybody is saying to load the library paths like I did below. But the error still remains.

I am able to deploy a dummy task (infinit loop) on mesos and it is running all right.

flink@cow-11:~/flink-1.9.0$ export MESOS_NATIVE_JAVA_LIBRARY="/home/felipe/workspace-vsc/mesos/build/src/.libs/libmesos.so"

flink@cow-11:~/flink-1.9.0$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/felipe/workspace-vsc/mesos/build/src/.libs/
flink@cow-11:~/flink-1.9.0$ PATH=$PATH:/home/felipe/workspace-vsc/mesos/build/bin/

flink-1.9.0$ sudo ./bin/mesos-appmaster.sh
Failed to load native Mesos library from /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib

Thanks,

Felipe

--