in the fat jar
In which way is the driver in the classpath?- fat jar?- in the nested /out folder in the slim jar?On Fri, Jun 5, 2015 at 7:23 PM, Flavio Pompermaier <[hidden email]> wrote:Actually I just need to load it in the main method (job manager) before calling any flink operation, I retrieve the records in a mysql table because they contain the path of files I'll need to read. Nothing more nothing less
On 5 Jun 2015 19:06, "Robert Metzger" <[hidden email]> wrote:Sure.So the DriverManager has a static variable called "registeredDrivers".When DriverManager.getConnection() is called, the method is looking up if an registered driver for that connection (in this case "mysql") is available.For drivers to be in that list, they have to register themselves using the DriverManager.registerDriver() method.Drivers can register themselves with a static constructor (which is executed when Java loads a class):public class PoolingDriver implements Driver {
/** Register myself with the {@link DriverManager}. */
static {
try {
DriverManager.registerDriver(new PoolingDriver());
} catch(Exception e) {
}
}To execute that driver registration, you need to do:Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver");because then Java is loading the class and executing the static constructor which is registering the driver at the connection manager.When executing Flink locally, you are using only one JVM. By calling the MySQL driver manually in the main() method of your flink job, you are registering the MySQL driver at the DriverManager of that JVM.However, when you run Flink in a distributed cluster, at the TaskManager JVMs, the MySQL driver is not loaded at the DriverManager there.Therefore, you have to make sure that Class.forName("org.datanucleus.store.rdbms.datasource.dbcp.PoolingDriver"); (this is not the correct class for the MySQL driver)has been called.One approach to do that is to call Class.forName() in the open() method of your function.Best,RobertOn Fri, Jun 5, 2015 at 6:54 PM, Flavio Pompermaier <[hidden email]> wrote:HI Robert,
In the main method I connect to a mysql table that acts as a data-source repository that I use to know which dataset I need to load. All mysql classes are present in the shaded jar.
Could you explain a little bit more in detail the solution to fix this problem please? Sorry but I didn't understand it :(Thanks,
FlavioOn 5 Jun 2015 18:33, "Robert Metzger" <[hidden email]> wrote:Hi Stefano,I doubt that there are conflicting dependencies because Flink does not contain MySQL dependencies.Are you using Flink's JDBCInputFormat or custom code?For drivers to register at java.sql's DriverManager, their classes need to be loaded first. To load a class, you need to call Class.forName("<classname>");
Maybe you are loading the class in the application's main() method (thats why it is working from eclipse) but not on the cluster instances which are supposed to read the data.On Fri, Jun 5, 2015 at 5:16 PM, Stefano Bortoli <[hidden email]> wrote:StefanoSaluti,Hi Robert,I answer on behalf of Flavio. He told me the driver jar was included. Smells lik class-loading issue due to 'conflicting' dependencies. Is it possible?2015-06-05 16:24 GMT+02:00 Robert Metzger <[hidden email]>:Hi,is the MySQL driver part of the Jar file that you've build?On Fri, Jun 5, 2015 at 4:11 PM, Flavio Pompermaier <[hidden email]> wrote:Hi to all,I'm using a fresh build of flink-0.9-SNAPSHOT and in my flink job I set up a mysql connection.When I run the job from Eclipse everything is fine,while when running the job from the Web UI I get the following exception:
java.sql.SQLException: No suitable driver found for jdbc:mysql:/localhost:3306/mydb?autoReconnect=true
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)How can I fix that?Best,Flavio
| Free forum by Nabble | Edit this page |