Error running an hadoop job from web interface
Locked
Error running an hadoop job from web interface
 
|
Hi to all,
I'm trying to run a job from the web interface but I get this error: java.lang.RuntimeException: java.io.FileNotFoundException: JAR entry core-site.xml not found in /tmp/webclient-jobs/EntitonsJsonizer.jar at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2334) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2187) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2104) at org.apache.hadoop.conf.Configuration.get(Configuration.java:853) at org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2088) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:446) at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:175) at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:156) at it.okkam.flink.entitons.io.utils.ParquetThriftEntitons.readEntitons(ParquetThriftEntitons.java:42) at it.okkam.flink.entitons.io.utils.ParquetThriftEntitons.readEntitonsWithId(ParquetThriftEntitons.java:73) at org.okkam.entitons.EntitonsJsonizer.readAtomQuads(EntitonsJsonizer.java:235) at org.okkam.entitons.EntitonsJsonizer.main(EntitonsJsonizer.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395) at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:80) at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:220) at org.apache.flink.client.CliFrontend.info(CliFrontend.java:412) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:977) at org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:171) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: JAR entry core-site.xml not found in /tmp/webclient-jobs/EntitonsJsonizer.jar at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:140) at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150) at java.net.URL.openStream(URL.java:1037) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2163) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2234) I checked the jar and it contains the core-site.xml file..Am I forced to configure the hadoop classpath in my Flink cluster config files? Best, Flavio |
Re: Error running an hadoop job from web interface
|
|
Hi Flavio, Which version of Flink are you using?On Fri, Oct 23, 2015 at 2:45 PM, Flavio Pompermaier <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
0.10-snapshot On 23 Oct 2015 16:09, "Maximilian Michels" <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
Could you try submitting the job from the command-line and see if it works? Thanks,On Fri, Oct 23, 2015 at 4:42 PM, Flavio Pompermaier <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
could you write ne the command please?I'm not in the office right now.. On 23 Oct 2015 17:10, "Maximilian Michels" <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
What I normally do is to does it make sense?java -cp MYUBERJAR.jar my.package.mainclass 2015-10-23 17:22 GMT+02:00 Flavio Pompermaier <[hidden email]>:
|
Re: Error running an hadoop job from web interface
|
|
./bin/flink run /path/to/jar arguments or ./bin/flink run -c MainClass /path/to/jar arguments On Fri, Oct 23, 2015 at 5:50 PM, Stefano Bortoli <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
Running from the shell everything works..is it a problem of classloaders hierarchy in the webapp?
On Fri, Oct 23, 2015 at 5:53 PM, Maximilian Michels <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
Correct. I'll fix it today. Cheers,On Mon, Oct 26, 2015 at 9:08 AM, Flavio Pompermaier <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
Now that I've recompiled flink and restarted the web-client everything works fine.
However, when I flag the job I want to run I see parallelism 1 in the right panel, but when I click on "Run Job" button + show optimizer plan flagged I see parallelism 36. Is that a bug of the first preview? On Mon, Oct 26, 2015 at 10:01 AM, Maximilian Michels <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
Did you set the default parallelism of the cluster to 36? This is because the plan gets optimized against the cluster configuration when you try to run the uploaded program. Before, it doesn't do any optimization. This might not be very intuitive. We should probably change that. On Mon, Oct 26, 2015 at 2:03 PM, Flavio Pompermaier <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
No, I just use the default parallelism
On Mon, Oct 26, 2015 at 3:05 PM, Maximilian Michels <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
That's odd. Does it also execute with parallelism 36 then? On Mon, Oct 26, 2015 at 3:06 PM, Flavio Pompermaier <[hidden email]> wrote:
|
Re: Error running an hadoop job from web interface
|
|
Yes, it's only the first job preview that has a wrong parallelism. The optimized plan and the job execution work just fine.
On Mon, Oct 26, 2015 at 3:11 PM, Maximilian Michels <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |