JDBCOutputFormat dependency loading error

[Flavio Pompermaier]

Hi to all, 

I'm trying to run my DataSet job on Flink 1.11.0 and I'm connecting toward Mariadb in my code.

I've put the mariadb-java-client-2.6.0.jar in the lib directory and in the pom.xml I set that dependency as provided. The code runs successfully from the Ide but when I try to run the code on the cluster I get the following error:

Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:315) ~[?:?]
at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider.getConnection(SimpleJdbcConnectionProvider.java:52) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.establishConnection(AbstractJdbcOutputFormat.java:66) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.open(AbstractJdbcOutputFormat.java:59) ~myApp.jar:?]
at org.apache.flink.api.java.io.jdbc.JDBCOutputFormat.open(JDBCOutputFormat.java:82) ~myApp.jar:?]
at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:834) ~[?:?]

What should I do?

Thanks in advance,

Flavio

[Till Rohrmann]

Hi Flavio,

I am not a JDBC expert but it looks as if you try to load com.mysql.cj.jdbc.Driver which is not contained in mariadb-java-client-2.6.0.jar. mariadb-java-client-2.6.0.jar only contains org/mariadb/jdbc/Driver.class. com.mysql.cj.jdbc.Driver can be found in mysql-connector-java.jar, though.

Hence I believe that you are missing some dependencies in your user jar to make your job run. Please check from where com.mysql.cj.jdbc.Driver is being loaded when running the job from the IDE.

Cheers,

Till

On Fri, Jul 31, 2020 at 4:55 PM Flavio Pompermaier <[hidden email]> wrote:

Hi to all, 

I'm trying to run my DataSet job on Flink 1.11.0 and I'm connecting toward Mariadb in my code.

I've put the mariadb-java-client-2.6.0.jar in the lib directory and in the pom.xml I set that dependency as provided. The code runs successfully from the Ide but when I try to run the code on the cluster I get the following error:

Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:315) ~[?:?]
at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider.getConnection(SimpleJdbcConnectionProvider.java:52) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.establishConnection(AbstractJdbcOutputFormat.java:66) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.open(AbstractJdbcOutputFormat.java:59) ~myApp.jar:?]
at org.apache.flink.api.java.io.jdbc.JDBCOutputFormat.open(JDBCOutputFormat.java:82) ~myApp.jar:?]
at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:834) ~[?:?]

What should I do?

Thanks in advance,

Flavio

[Flavio Pompermaier]

Yes Till, I was figuring out that I was using 2 different connectors and I forgot the mysql jar..I was going to test and tell if the problem was solved!

On Mon, Aug 3, 2020 at 10:50 AM Till Rohrmann <[hidden email]> wrote:

Hi Flavio,

I am not a JDBC expert but it looks as if you try to load com.mysql.cj.jdbc.Driver which is not contained in mariadb-java-client-2.6.0.jar. mariadb-java-client-2.6.0.jar only contains org/mariadb/jdbc/Driver.class. com.mysql.cj.jdbc.Driver can be found in mysql-connector-java.jar, though.

Hence I believe that you are missing some dependencies in your user jar to make your job run. Please check from where com.mysql.cj.jdbc.Driver is being loaded when running the job from the IDE.

Cheers,

Till

On Fri, Jul 31, 2020 at 4:55 PM Flavio Pompermaier <[hidden email]> wrote:

Hi to all, 

I'm trying to run my DataSet job on Flink 1.11.0 and I'm connecting toward Mariadb in my code.

I've put the mariadb-java-client-2.6.0.jar in the lib directory and in the pom.xml I set that dependency as provided. The code runs successfully from the Ide but when I try to run the code on the cluster I get the following error:

Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:315) ~[?:?]
at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider.getConnection(SimpleJdbcConnectionProvider.java:52) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.establishConnection(AbstractJdbcOutputFormat.java:66) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.open(AbstractJdbcOutputFormat.java:59) ~myApp.jar:?]
at org.apache.flink.api.java.io.jdbc.JDBCOutputFormat.open(JDBCOutputFormat.java:82) ~myApp.jar:?]
at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:834) ~[?:?]

What should I do?

Thanks in advance,

Flavio

[Till Rohrmann]

Glad to hear it! 

On Mon, Aug 3, 2020 at 11:59 AM Flavio Pompermaier <[hidden email]> wrote:

Yes Till, I was figuring out that I was using 2 different connectors and I forgot the mysql jar..I was going to test and tell if the problem was solved!

On Mon, Aug 3, 2020 at 10:50 AM Till Rohrmann <[hidden email]> wrote:

Hi Flavio,

I am not a JDBC expert but it looks as if you try to load com.mysql.cj.jdbc.Driver which is not contained in mariadb-java-client-2.6.0.jar. mariadb-java-client-2.6.0.jar only contains org/mariadb/jdbc/Driver.class. com.mysql.cj.jdbc.Driver can be found in mysql-connector-java.jar, though.

Hence I believe that you are missing some dependencies in your user jar to make your job run. Please check from where com.mysql.cj.jdbc.Driver is being loaded when running the job from the IDE.

Cheers,

Till

On Fri, Jul 31, 2020 at 4:55 PM Flavio Pompermaier <[hidden email]> wrote:

Hi to all, 

I'm trying to run my DataSet job on Flink 1.11.0 and I'm connecting toward Mariadb in my code.

I've put the mariadb-java-client-2.6.0.jar in the lib directory and in the pom.xml I set that dependency as provided. The code runs successfully from the Ide but when I try to run the code on the cluster I get the following error:

Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:315) ~[?:?]
at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider.getConnection(SimpleJdbcConnectionProvider.java:52) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.establishConnection(AbstractJdbcOutputFormat.java:66) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.open(AbstractJdbcOutputFormat.java:59) ~myApp.jar:?]
at org.apache.flink.api.java.io.jdbc.JDBCOutputFormat.open(JDBCOutputFormat.java:82) ~myApp.jar:?]
at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:834) ~[?:?]

What should I do?

Thanks in advance,

Flavio

[Flavio Pompermaier]

Yes, the problem indeed was mine (2 different connectors for mariadb, both mysql and mariadb-client), Sorry for the confusion

On Mon, Aug 3, 2020 at 12:26 PM Till Rohrmann <[hidden email]> wrote:

Glad to hear it! 

On Mon, Aug 3, 2020 at 11:59 AM Flavio Pompermaier <[hidden email]> wrote:

Yes Till, I was figuring out that I was using 2 different connectors and I forgot the mysql jar..I was going to test and tell if the problem was solved!

On Mon, Aug 3, 2020 at 10:50 AM Till Rohrmann <[hidden email]> wrote:

Hi Flavio,

I am not a JDBC expert but it looks as if you try to load com.mysql.cj.jdbc.Driver which is not contained in mariadb-java-client-2.6.0.jar. mariadb-java-client-2.6.0.jar only contains org/mariadb/jdbc/Driver.class. com.mysql.cj.jdbc.Driver can be found in mysql-connector-java.jar, though.

Hence I believe that you are missing some dependencies in your user jar to make your job run. Please check from where com.mysql.cj.jdbc.Driver is being loaded when running the job from the IDE.

Cheers,

Till

On Fri, Jul 31, 2020 at 4:55 PM Flavio Pompermaier <[hidden email]> wrote:

Hi to all, 

I'm trying to run my DataSet job on Flink 1.11.0 and I'm connecting toward Mariadb in my code.

I've put the mariadb-java-client-2.6.0.jar in the lib directory and in the pom.xml I set that dependency as provided. The code runs successfully from the Ide but when I try to run the code on the cluster I get the following error:

Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:61) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:315) ~[?:?]
at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider.getConnection(SimpleJdbcConnectionProvider.java:52) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.establishConnection(AbstractJdbcOutputFormat.java:66) ~myApp.jar:?]
at org.apache.flink.connector.jdbc.internal.AbstractJdbcOutputFormat.open(AbstractJdbcOutputFormat.java:59) ~myApp.jar:?]
at org.apache.flink.api.java.io.jdbc.JDBCOutputFormat.open(JDBCOutputFormat.java:82) ~myApp.jar:?]
at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) ~[flink-dist_2.12-1.11.0.jar:1.11.0]
at java.lang.Thread.run(Thread.java:834) ~[?:?]

What should I do?

Thanks in advance,

Flavio