flink run jar throw java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

Posted by alex wang on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/flink-run-jar-throw-java-lang-ClassNotFoundException-com-mysql-jdbc-Driver-tp30757.html

Hello everyone, I am a newbie.
I am learning the flink-sql-submit project. From @Jark Wu  :https://github.com/wuchong/flink-sql-submit

My local environment is:
1. flink1.9.0 standalone
2. kafka_2.11-2.2.0 single

I configured Flink Connectors and Formats jars to $FLINK_HOME/lib .
Reference: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#connectors

Then I run flink-sql-submit , sh run.sh q1
Throw  java.lang.ClassNotFoundException: com.mysql.jdbc.Driver.

My question is: 

I configured mysql-connector-java in the pom.xml file, mvn build jar include com.mysql.jdbc.Driver. 

Why is this error still reported? I put the jar package in $FLINK_HOME/lib and the problem can be solved.

Do you need to put these jars in $FLINK_HOME/lib when the project relies on too many jar packages?

If I don't put mysql-connector-java.jar in $FLINK_HOME/lib, how can I solve this problem?

Can @Jark Wu  give me some advice? Or can someone give me some advice? Thank you.

1. pom.xml

<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
</dependency>

2. mvn clean; mvn package

$ ll -rth target                                                                                                                                [±master ●]
total 32312
drwxr-xr-x  3 alex  staff    96B Oct 30 11:32 generated-sources
drwxr-xr-x  5 alex  staff   160B Oct 30 11:32 classes
drwxr-xr-x  3 alex  staff    96B Oct 30 11:32 maven-archiver
-rw-r--r--  1 alex  staff   7.2M Oct 30 11:32 flink-sql-submit-1.0-SNAPSHOT.jar
-rw-r--r--  1 alex  staff   8.2M Oct 30 11:32 flink-sql-submit.jar

3. flink-sql-submit.jar include java.sql.Driver  

" zip.vim version v28
" Browsing zipfile /Users/alex/IdeaProjects/alex/flink_learn/flink-sql-submit/target/flink-sql-submit.jar
" Select a file with cursor and press ENTER

META-INF/MANIFEST.MF
META-INF/
q1.sql
user_behavior.log
com/
com/github/
com/github/wuchong/
com/github/wuchong/sqlsubmit/
com/github/wuchong/sqlsubmit/SqlSubmit$1.class
com/github/wuchong/sqlsubmit/SqlSubmit.class
com/github/wuchong/sqlsubmit/SourceGenerator.class
com/github/wuchong/sqlsubmit/cli/
com/github/wuchong/sqlsubmit/cli/SqlCommandParser$SqlCommandCall.class
com/github/wuchong/sqlsubmit/cli/SqlCommandParser.class
com/github/wuchong/sqlsubmit/cli/SqlCommandParser$SqlCommand.class
com/github/wuchong/sqlsubmit/cli/CliOptions.class
com/github/wuchong/sqlsubmit/cli/CliOptionsParser.class
META-INF/maven/
META-INF/maven/com.github.wuchong/
META-INF/maven/com.github.wuchong/flink-sql-submit/
META-INF/maven/com.github.wuchong/flink-sql-submit/pom.xml
META-INF/maven/com.github.wuchong/flink-sql-submit/pom.properties
META-INF/services/
META-INF/services/java.sql.Driver
com/mysql/
com/mysql/fabric/
com/mysql/fabric/FabricCommunicationException.class
com/mysql/fabric/FabricConnection.class
com/mysql/fabric/FabricStateResponse.class
com/mysql/fabric/HashShardMapping$ReverseShardIndexSorter.class
com/mysql/fabric/HashShardMapping.class
com/mysql/fabric/RangeShardMapping$RangeShardIndexSorter.class
com/mysql/fabric/RangeShardMapping.class
com/mysql/fabric/Response.class
com/mysql/fabric/Server.class
com/mysql/fabric/ServerGroup.class
com/mysql/fabric/ServerMode.class
com/mysql/fabric/ServerRole.class
etc ...

 

$FLINK_DIR/bin/flink run -d -p 3 target/flink-sql-submit.jar -w "${PROJECT_DIR}"/src/main/resources/ -f "$1".sql  

Eerror:
2019-10-30 10:27:35
java.lang.IllegalArgumentException: JDBC driver class not found.
At org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.open(JDBCUpsertOutputFormat.java:112)
At org.apache.flink.api.java.io.jdbc.JDBCUpsertSinkFunction.open(JDBCUpsertSinkFunction.java:42)
At org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
At org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
At org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
At org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:532)
At org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:396)
At org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
At org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
At java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
At java.net.URLClassLoader.findClass(URLClassLoader.java:381)
At java.lang.ClassLoader.loadClass(ClassLoader.java:424)
At sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
At java.lang.ClassLoader.loadClass(ClassLoader.java:357)
At java.lang.Class.forName0(Native Method)
At java.lang.Class.forName(Class.java:264)
At org.apache.flink.api.java.io.jdbc.AbstractJDBCOutputFormat.establishConnection(AbstractJDBCOutputFormat.java:66)
At org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.open(JDBCUpsertOutputFormat.java:99)
... 9 more


--
Best