Cannot instantiate Mysql connection

[Flavio Pompermaier]

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[rmetzger0]

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Stefano Bortoli]

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[rmetzger0]

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Flavio Pompermaier]

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[rmetzger0]

Sure.

So the DriverManager has a static variable called "registeredDrivers".

When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.

For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.

Drivers can register themselves with a static constructor (which is executed when Java loads a class):

public class PoolingDriver implements Driver {
    /** Register myself with the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {
        }
    }

To execute that driver registration, you need to do:

Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");

because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.

Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)

has been called.

One approach to do that is to call Class.forName() in the open() method of your function.

Best,

Robert

On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Flavio Pompermaier]

Actually I just need to load it in the main method (job manager) before calling any flink operation, I retrieve the records in a mysql table because they contain the path of files I'll need to read. Nothing more nothing less

On 5 Jun 2015 19:06, "Robert Metzger" <[hidden email]> wrote:

Sure.

So the DriverManager has a static variable called "registeredDrivers".

When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.

For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.

Drivers can register themselves with a static constructor (which is executed when Java loads a class):

public class PoolingDriver implements Driver {
    /** Register myself with the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {
        }
    }

To execute that driver registration, you need to do:

Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");

because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.

Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)

has been called.

One approach to do that is to call Class.forName() in the open() method of your function.

Best,

Robert

On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Stephan Ewen]

In which way is the driver in the classpath?

 - fat jar?

 - in the nested /out folder in the slim jar?

On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <[hidden email]> wrote:

Actually I just need to load it in the main method (job manager) before calling any flink operation, I retrieve the records in a mysql table because they contain the path of files I'll need to read. Nothing more nothing less

On 5 Jun 2015 19:06, "Robert Metzger" <[hidden email]> wrote:

Sure.

So the DriverManager has a static variable called "registeredDrivers".

When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.

For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.

Drivers can register themselves with a static constructor (which is executed when Java loads a class):

public class PoolingDriver implements Driver {
    /** Register myself with the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {
        }
    }

To execute that driver registration, you need to do:

Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");

because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.

Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)

has been called.

One approach to do that is to call Class.forName() in the open() method of your function.

Best,

Robert

On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Flavio Pompermaier]

in the fat jar

On 5 Jun 2015 19:28, "Stephan Ewen" <[hidden email]> wrote:

In which way is the driver in the classpath?

 - fat jar?

 - in the nested /out folder in the slim jar?

On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <[hidden email]> wrote:

Actually I just need to load it in the main method (job manager) before calling any flink operation, I retrieve the records in a mysql table because they contain the path of files I'll need to read. Nothing more nothing less

On 5 Jun 2015 19:06, "Robert Metzger" <[hidden email]> wrote:

Sure.

So the DriverManager has a static variable called "registeredDrivers".

When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.

For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.

Drivers can register themselves with a static constructor (which is executed when Java loads a class):

public class PoolingDriver implements Driver {
    /** Register myself with the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {
        }
    }

To execute that driver registration, you need to do:

Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");

because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.

Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)

has been called.

One approach to do that is to call Class.forName() in the open() method of your function.

Best,

Robert

On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Stephan Ewen]

Can you manually load the driver class, with "Class.forName(...)", or does that yield a "ClassNotFoundException" ?

On Fri, Jun 5, 2015 at 11:10 PM, Flavio Pompermaier <[hidden email]> wrote:

in the fat jar

On 5 Jun 2015 19:28, "Stephan Ewen" <[hidden email]> wrote:

In which way is the driver in the classpath?

 - fat jar?

 - in the nested /out folder in the slim jar?

On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <[hidden email]> wrote:

Actually I just need to load it in the main method (job manager) before calling any flink operation, I retrieve the records in a mysql table because they contain the path of files I'll need to read. Nothing more nothing less

On 5 Jun 2015 19:06, "Robert Metzger" <[hidden email]> wrote:

Sure.

So the DriverManager has a static variable called "registeredDrivers".

When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.

For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.

Drivers can register themselves with a static constructor (which is executed when Java loads a class):

public class PoolingDriver implements Driver {
    /** Register myself with the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {
        }
    }

To execute that driver registration, you need to do:

Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");

because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.

Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)

has been called.

One approach to do that is to call Class.forName() in the open() method of your function.

Best,

Robert

On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Flavio Pompermaier]

Sorry for the late response but I was on vacation the last 2 weeks..

Calling Class.forName("com.mysql.jdbc.Driver") in the main() of my class made the things work!

Thanks for the support,

Flavio

On Fri, Jun 5, 2015 at 11:13 PM, Stephan Ewen <[hidden email]> wrote:

Can you manually load the driver class, with "Class.forName(...)", or does that yield a "ClassNotFoundException" ?

On Fri, Jun 5, 2015 at 11:10 PM, Flavio Pompermaier <[hidden email]> wrote:

in the fat jar

On 5 Jun 2015 19:28, "Stephan Ewen" <[hidden email]> wrote:

In which way is the driver in the classpath?

 - fat jar?

 - in the nested /out folder in the slim jar?

On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <[hidden email]> wrote:

Actually I just need to load it in the main method (job manager) before calling any flink operation, I retrieve the records in a mysql table because they contain the path of files I'll need to read. Nothing more nothing less

On 5 Jun 2015 19:06, "Robert Metzger" <[hidden email]> wrote:

Sure.

So the DriverManager has a static variable called "registeredDrivers".

When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.

For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.

Drivers can register themselves with a static constructor (which is executed when Java loads a class):

public class PoolingDriver implements Driver {
    /** Register myself with the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {
        }
    }

To execute that driver registration, you need to do:

Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");

because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.

Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)

has been called.

One approach to do that is to call Class.forName() in the open() method of your function.

Best,

Robert

On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio

[Stephan Ewen]

Good to hear it works.

Libraries, class-loading, and initialization seems to be one of the things that remains tricky once one switches to distributed processed.

On Thu, Jun 25, 2015 at 10:58 AM, Flavio Pompermaier <[hidden email]> wrote:

Sorry for the late response but I was on vacation the last 2 weeks..

Calling Class.forName("com.mysql.jdbc.Driver") in the main() of my class made the things work!

Thanks for the support,

Flavio

On Fri, Jun 5, 2015 at 11:13 PM, Stephan Ewen <[hidden email]> wrote:

Can you manually load the driver class, with "Class.forName(...)", or does that yield a "ClassNotFoundException" ?

On Fri, Jun 5, 2015 at 11:10 PM, Flavio Pompermaier <[hidden email]> wrote:

in the fat jar

On 5 Jun 2015 19:28, "Stephan Ewen" <[hidden email]> wrote:

In which way is the driver in the classpath?

 - fat jar?

 - in the nested /out folder in the slim jar?

On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <[hidden email]> wrote:

Actually I just need to load it in the main method (job manager) before calling any flink operation, I retrieve the records in a mysql table because they contain the path of files I'll need to read. Nothing more nothing less

On 5 Jun 2015 19:06, "Robert Metzger" <[hidden email]> wrote:

Sure.

So the DriverManager has a static variable called "registeredDrivers".

When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.

For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.

Drivers can register themselves with a static constructor (which is executed when Java loads a class):

public class PoolingDriver implements Driver {
    /** Register myself with the {@link DriverManager}. */
    static {
        try {
            DriverManager.registerDriver(new PoolingDriver());
        } catch(Exception e) {
        }
    }

To execute that driver registration, you need to do:

Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");

because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.

When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.

However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.

Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)

has been called.

One approach to do that is to call Class.forName() in the open() method of your function.

Best,

Robert

On Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:

HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(

Thanks,
Flavio

On 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:

Hi Stefano,

I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.

Are you using Flink's JDBCInputFormat or custom code?

For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");

Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.

On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:

Hi Robert,

I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies.  Is it possible?

Saluti,

Stefano

2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

is the MySQL driver part of the Jar file that you've build?

On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.

When I run the job from Eclipse everything is fine,

while when running the job from the Web UI I get the following exception:

java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)

How can I fix that?

Best,

Flavio