RE: Cluster installation gives java.lang.NoClassDefFoundError for everything - detailed debugging

[Camelia Elena Ciolac]

 Hello,

As promised, I come back with debugging details.
So:

******* In start-cluster.sh , the following echo's

echo "start-cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $HOSTLIST
echo "start-cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo $FLINK_BIN_DIR
echo "start-cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"

# cluster mode, bring up job manager locally and a task manager on every slave host
"$FLINK_BIN_DIR"/jobmanager.sh start cluster batch

==>  gave:

start-cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster/flink-0.9.1/conf/slaves
start-cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster/flink-0.9.1/bin
start-cluster ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


******* In jobmanager.sh , the following echo's

echo "Starting Job Manager"
        echo "jobmanager ~~~~~~~~~~~~~~~~~~~~~"
        pwd
        echo "jobmanager ~~~~~~~~~~~~~~~~~~~~~"
        echo ${FLINK_ENV_JAVA_OPTS}
        echo "jobmanager ~~~~~~~~~~~~~~~~~~~~~"
        echo $FLINK_JM_CLASSPATH
        echo "jobmanager ~~~~~~~~~~~~~~~~~~~~~"
        echo $INTERNAL_HADOOP_CLASSPATHS
        echo "jobmanager ~~~~~~~~~~~~~~~~~~~~"
        echo $FLINK_CONF_DIR
        echo "~~~~~~~~~~~~~~~~~~~~"
        $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_JM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.runtime.jobmanager.JobManager --configDir "$FLINK_CONF_DIR" --executionMode $EXECUTIONMODE --streamingMode "$STREAMINGMODE" > "$out" 2>&1 < /dev/null &

     

==> gave:

jobmanager ~~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster
jobmanager ~~~~~~~~~~~~~~~~~~~~~

jobmanager ~~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster/flink-0.9.1/lib/flink-dist-0.9.1.jar:/users/camelia/thecluster/flink-0.9.1/lib/flink-python-0.9.1.jar
jobmanager ~~~~~~~~~~~~~~~~~~~~~
::
jobmanager ~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster/flink-0.9.1/conf
~~~~~~~~~~~~~~~~~~~~

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/runtime/jobmanager/JobManager
Caused by: java.lang.ClassNotFoundException: org.apache.flink.runtime.jobmanager.JobManager
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.flink.runtime.jobmanager.JobManager. Program will exit.



******* In taskmanager.sh, the following echo's

echo Starting task manager on host $HOSTNAME
        echo "taskmanager ~~~~~~~~~~~~~~~~~~~~~"
        pwd
        echo "taskmanager ~~~~~~~~~~~~~~~~~~~~~"
        echo ${FLINK_ENV_JAVA_OPTS}
        echo "taskmanager ~~~~~~~~~~~~~~~~~~~~~"
        echo $FLINK_JM_CLASSPATH
        echo "taskmanager ~~~~~~~~~~~~~~~~~~~~~"
        echo $INTERNAL_HADOOP_CLASSPATHS
        echo "taskmanager ~~~~~~~~~~~~~~~~~~~~"
        echo $FLINK_CONF_DIR
        echo "~~~~~~~~~~~~~~~~~~~~"
        $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.runtime.taskmanager.TaskManager --configDir "$FLINK_CONF_DIR" --streamingMode "$STREAMINGMODE" > "$out" 2>&1 < /dev/null &

==> gave:

taskmanager ~~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster
taskmanager ~~~~~~~~~~~~~~~~~~~~~

taskmanager ~~~~~~~~~~~~~~~~~~~~~

taskmanager ~~~~~~~~~~~~~~~~~~~~~
::
taskmanager ~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster/flink-0.9.1/conf
~~~~~~~~~~~~~~~~~~~~

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/runtime/taskmanager/TaskManager
Caused by: java.lang.ClassNotFoundException: org.apache.flink.runtime.taskmanager.TaskManager
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.flink.runtime.taskmanager.TaskManager. Program will exit.



******* In flink , the following echo's

echo "flink ~~~~~~~~~~~~~~~~~~~~~"
pwd
echo "flink ~~~~~~~~~~~~~~~~~~~~~"
echo $CC_CLASSPATH
echo "flink ~~~~~~~~~~~~~~~~~~~~~"
echo $INTERNAL_HADOOP_CLASSPATHS
echo "~~~~~~~~~~~~~~~~~~~~"
$JAVA_RUN $JVM_ARGS "${log_setting[@]}" -classpath "`manglePathList "$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`" org.apache.flink.client.CliFrontend "$@"


==> gave:

flink ~~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster
flink ~~~~~~~~~~~~~~~~~~~~~
/users/camelia/thecluster/flink-0.9.1/lib/flink-dist-0.9.1.jar:/users/camelia/thecluster/flink-0.9.1/lib/flink-python-0.9.1.jar
flink ~~~~~~~~~~~~~~~~~~~~~
::
~~~~~~~~~~~~~~~~~~~~
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/client/CliFrontend
Caused by: java.lang.ClassNotFoundException: org.apache.flink.client.CliFrontend
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.flink.client.CliFrontend. Program will exit.


I hope that now you have a better image of the issue and I hope you can provide some guidance to solve it.
I mention that I use the simplest scenario possible, just 2 nodes (1 master and 1 slave) without Hadoop for this test.
Also, I have a hostname command inside the script that shows me that I'm on the master node when running these scripts.

Thank you very much!
Camelia

---

From: Camelia Elena Ciolac
Sent: Wednesday, November 11, 2015 9:01 AM
To: [hidden email]
Subject: RE: Cluster installation gives java.lang.NoClassDefFoundError for everything

Good morning,

Thank you Stephan!

I keep on testing and in the meantime I'm wondering if the Java version on the cluster may be part of the issue:
java version "1.6.0_36"
OpenJDK Runtime Environment (IcedTea6 1.13.8) (rhel-1.13.8.1.el6_7-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)
?

The cluster has shared network file system, so no problem in this aspect.
I keep on trying and I've put some echo's for debugging in the Flink start scripts.

Best regards,
Camelia

---

From: Camelia Elena Ciolac
Sent: Monday, November 09, 2015 2:32 PM
To: [hidden email]
Subject: Cluster installation gives java.lang.NoClassDefFoundError for everything

Hello,

I am configuring Flink to run on a cluster with NFS.
I have the Flink 0.9.1 distribution in some path in NFS and I added that path in ~/.bashrc as FLINK_HOME, and also included the $FLINK_HOME/lib folder to $PATH.
I have the slaves file and the yaml file configured correctly with the nodes involved.

Still, when I run the following command I get the following errors:

$FLINK_HOME/bin/start-cluster.sh

(tried also
cd $FLINK_HOME
./bin/start-cluster.sh)

--> in /log/flink-theuser-jobmanager-thenode.out
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/runtime/jobmanager/JobManager
Caused by: java.lang.ClassNotFoundException: org.apache.flink.runtime.jobmanager.JobManager
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.flink.runtime.jobmanager.JobManager. Program will exit.

-->in /log/flink-theuser-taskmanager-theslavenode.out
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/runtime/taskmanager/TaskManager
Caused by: java.lang.ClassNotFoundException: org.apache.flink.runtime.taskmanager.TaskManager
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.flink.runtime.taskmanager.TaskManager. Program will exit.


And lastly, in my script I have the run of a job which gives:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/client/CliFrontend
Caused by: java.lang.ClassNotFoundException: org.apache.flink.client.CliFrontend
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.flink.client.CliFrontend. Program will exit.

What could be the cause and which is the solution?

Looking forward to your answer as soon as possible.
Many thanks,
Camelia