|
|
Flink 1.11 will support Hadoop 3. EMR 6 requires Hadoop 3, that's why Flink was not included anymore. Amazon will add Flink back to EMR 6.0 soon. Hi Yang,
I am able to resolve the issue by removing Hadoop dependency as you mentioned.
1. Removed hadoop-common dependency and <groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-fs</artifactId>
</exclusion>
</exclusions>
</dependency>
2. After the above changes, I am able to submit job on yarn but facing issues with s3 plugin that I resolved by putting flink-s3-fs-hadoop-1.9.0.jar i n the plugins/s3-fs-hadoop directory.
Thanks for your support.
Any update when will flink.10 officially supported in EMR. Even in new EMR version(emr 6.0) flink has been removed.
Hello Yang,
I have attached my pom file and I did not see that I am using any Hadoop dependency. Can you please help me.
Hi aj,
From the logs you have provided, the hadoop version is still 2.4.1. Could you check the user jar(i.e. events-processor-1.0-SNAPSHOT.jar) have some hadoop classes? If it is, you need to exclude the hadoop dependency.
Best, Yang
Hello,
Please help me upgrade to 1.10 in AWS EMR.
Hi Yang,
I am attaching the logs for your reference, please help me what i am doing wrong.
Thanks, Anuj
Hi Anuj,
I think the exception you come across still because the hadoop version is 2.4.1. I have checked the hadoop code, the code line are exactly same. For 2.8.1, i also have checked the ruleParse. It could work.
/**
* A pattern for parsing a auth_to_local rule.
*/
private static final Pattern ruleParser =
Pattern.compile("\\s*((DEFAULT)|(RULE:\\[(\\d*):([^\\]]*)](\\(([^)]*)\\))?"+
"(s/([^/]*)/([^/]*)/(g)?)?))/?(L)?");
Could you share the jobmanager logs so that i could check the classpath and hadoop version?
Best, Yang
Hello Yang, My Hadoop version is Hadoop 3.2.1-amzn-0 and I have put in flink/lib. flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
then I am getting below error :
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1587983834922_0002/filecache/10/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "main" java.lang.IllegalArgumentException: Invalid rule: /L RULE:[2:$1@$0](.*@)s/@.*///L DEFAULT at org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:321) at org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:386) at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605) at org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:136) at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:109)
if I remove the flink-shaded-hadoop-2-uber-2.8.3-10.0.jar from lib then i get below error:
2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend - Classpath: /usr/lib/flink/lib/flink-table-blink_2.11-1.10.0.jar:/usr/lib/flink/lib/flink-table_2.11-1.10.0.jar:/usr/lib/flink/lib/log4j-1.2.17.jar:/usr/lib/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/lib/flink/lib/flink-dist_2.11-1.10.0.jar::/etc/hadoop/conf:/etc/hadoop/conf
2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend - --------------------------------------------------------------------------------
2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1568m
2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.yarn.conf.dir, /etc/hadoop/conf
2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region
2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first
2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIA52DD5QA5FC7HPKXG
2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ******
2020-04-27 16:59:37,305 WARN org.apache.flink.client.cli.CliFrontend - Could not load CLI class org.apache.flink.yarn.cli.FlinkYarnSessionCli.
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1076)
at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1030)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:957)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 5 more
2020-04-27 16:59:37,406 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2020-04-27 16:59:37,458 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2020-04-27 16:59:37,476 INFO org.apache.flink.runtime.security.modules.JaasModule - Jaas file will be created as /tmp/jaas-7054453135321774613.conf.
2020-04-27 16:59:37,480 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2020-04-27 16:59:37,481 INFO org.apache.flink.client.cli.CliFrontend - Running 'run' command.
2020-04-27 16:59:37,488 INFO org.apache.flink.client.cli.CliFrontend - Building program from JAR file
2020-04-27 16:59:37,488 ERROR org.apache.flink.client.cli.CliFrontend - Invalid command line arguments.
org.apache.flink.client.cli.CliArgsException: Could not build the program from JAR file.
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:203)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
Caused by: java.io.FileNotFoundException: JAR file does not exist: -ynm
at org.apache.flink.client.cli.CliFrontend.getJarFile(CliFrontend.java:719)
at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:695)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:200)
... 4 more
Can you please help.
Thanks, Anuj
Hi Anuj,
It seems that you are using hadoop version 2.4.1. I think "L" could not be supported in this version. Could you upgrade your hadoop version to 2.8 and have a try? If your YARN cluster version is 2.8+, then you could directly remove the flink-shaded-hadoop in your lib directory. Otherwise, you need to download the flink-shaded-hadoop with version 2.8 here[1].
Best, Yang
Hi Robert, attached the full application log file.
Thanks, Anuj
--
Thanks & Regards,
Anuj Jain
Mob. : +91- 8588817877
Skype : anuj.jain07
--
Thanks & Regards,
Anuj Jain
Mob. : +91- 8588817877
Skype : anuj.jain07
--
Thanks & Regards,
Anuj Jain
Mob. : +91- 8588817877
Skype : anuj.jain07
--
Thanks & Regards,
Anuj Jain
Mob. : +91- 8588817877
Skype : anuj.jain07
-- Thanks & Regards,
Anuj Jain
Mob. : +91- 8588817877
Skype : anuj.jain07
|
Locked
error.txt (51K)