Error submitting stand-alone Flink job to EMR YARN cluster
Locked
Error submitting stand-alone Flink job to EMR YARN cluster
 
|
I’m trying to submit a stand-alone Flink job to a YARN cluster running on EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container for the Job Manager, it fails. The error message
from the container is below. The command I’m using is:
$ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount ./flink-1.0.3/examples/streaming/WordCount.jar
I have tried adding log4j and slf4j libraries to the classpath using -C and that doesn’t help.
This does not happen on other YARN clusters I have that are not EMR nodes. And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to create a Flink cluster in the YARN cluster and then
use “flink run …” to submit the job to the Flink cluster.
Does anyone out there know how I could fix this?
Thanks in advance for any help you can give.
Error message in the jobmanager.err file:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:278) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.security.Groups.<init>(Groups.java:79) at org.apache.hadoop.security.Groups.<init>(Groups.java:74) at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) at org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64) at org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36) at org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 18 more
Software Engineer HERE Predictive Analytics
HERE Seattle
701 Pike St, Suite 2000, Seattle, WA 98101
|
Re: Error submitting stand-alone Flink job to EMR YARN cluster
|
|
I know this is really basic but have you verified that you're Flink lib folder contains log4j-1.2.17.jar? I imagine that's fine given the yarn-session.sh approach is working fine. What version of EMR are you running? What version of Flink? -Jamie On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <[hidden email]> wrote:
|
Re: Error submitting stand-alone Flink job to EMR YARN cluster
|
|
In reply to this post by Hanson, Bruce
Hi Bruce, I just spun up an EMR cluster and tried this out. Hadoop 2.7.2 and Flink 1.0.3. I ran the exact same command as you and everything works just fine. Please verify one thing, though. In your command you do not specify the path to the Flink executable, which means it's just getting picked up from your PATH. Is it possible you're running a flink-1.0.3 example program but using a different version of Flink? Try the following: -Jamie On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <[hidden email]> wrote:
|
Re: Error submitting stand-alone Flink job to EMR YARN cluster
|
|
You can also have a look at the YARN client logs, which should print which JARs are uploaded. The container logs should also log the class path. On Sun, Jul 3, 2016 at 6:04 PM, Jamie Grier <[hidden email]> wrote:
|
| Free forum by Nabble | Edit this page |