Hello All,
I am running Flink on AWS EMR, as currently the latest version available on EMR is 1.9.1 but I want to upgrade to 1.10.0. I tried to manually replace lib jars by downloading the 1.10.0 version but this is not working. I am getting the following exception when trying to submit a job on yarn. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "main" java.lang.IllegalArgumentException: Invalid rule: /L RULE:[2:$1@$0](.*@)s/@.*///L DEFAULT at org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:321) at org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:386) at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605) at org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:136) at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:109) End of LogType:jobmanager.err |
Hey Anuj, can you post the "header" of the jobmanager log, I'm interested in seeing the classpath of your jobmanager. Most likely, there's a mixup in your dependency versions in your classpath. On Tue, Apr 7, 2020 at 8:08 AM aj <[hidden email]> wrote:
|
error.txt (51K) Download Attachment |
Hi Anuj, It seems that you are using hadoop version 2.4.1. I think "L" could not be supported in this version. Could you upgrade your hadoop version to 2.8 and have a try? If your YARN cluster version is 2.8+, then you could directly remove the flink-shaded-hadoop in your lib directory. Otherwise, you need to download the flink-shaded-hadoop with version 2.8 here[1]. Best, Yang aj <[hidden email]> 于2020年4月11日周六 上午4:21写道: |
Hello Yang, My Hadoop version is Hadoop 3.2.1-amzn-0 and I have put in flink/lib. flink-shaded-hadoop-2-uber-2.8.3-10.0.jar then I am getting below error : SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/appcache/application_1587983834922_0002/filecache/10/slf4j-log4j12-1.7.15.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "main" java.lang.IllegalArgumentException: Invalid rule: /L RULE:[2:$1@$0](.*@)s/@.*///L DEFAULT at org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:321) at org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:386) at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605) at org.apache.flink.yarn.entrypoint.YarnEntrypointUtils.logYarnEnvironmentInformation(YarnEntrypointUtils.java:136) at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:109) if I remove the flink-shaded-hadoop-2-uber-2.8.3-10.0.jar from lib then i get below error: 2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend - Classpath: /usr/lib/flink/lib/flink-table-blink_2.11-1.10.0.jar:/usr/lib/flink/lib/flink-table_2.11-1.10.0.jar:/usr/lib/flink/lib/log4j-1.2.17.jar:/usr/lib/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/lib/flink/lib/flink-dist_2.11-1.10.0.jar::/etc/hadoop/conf:/etc/hadoop/conf 2020-04-27 16:59:37,293 INFO org.apache.flink.client.cli.CliFrontend - -------------------------------------------------------------------------------- 2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m 2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1568m 2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1 2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1 2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.yarn.conf.dir, /etc/hadoop/conf 2020-04-27 16:59:37,300 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.hadoop.conf.dir, /etc/hadoop/conf 2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region 2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.resolve-order, parent-first 2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, AKIA52DD5QA5FC7HPKXG 2020-04-27 16:59:37,301 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ****** 2020-04-27 16:59:37,305 WARN org.apache.flink.client.cli.CliFrontend - Could not load CLI class org.apache.flink.yarn.cli.FlinkYarnSessionCli. java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1076) at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1030) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:957) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ... 5 more 2020-04-27 16:59:37,406 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. 2020-04-27 16:59:37,458 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. 2020-04-27 16:59:37,476 INFO org.apache.flink.runtime.security.modules.JaasModule - Jaas file will be created as /tmp/jaas-7054453135321774613.conf. 2020-04-27 16:59:37,480 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. 2020-04-27 16:59:37,481 INFO org.apache.flink.client.cli.CliFrontend - Running 'run' command. 2020-04-27 16:59:37,488 INFO org.apache.flink.client.cli.CliFrontend - Building program from JAR file 2020-04-27 16:59:37,488 ERROR org.apache.flink.client.cli.CliFrontend - Invalid command line arguments. org.apache.flink.client.cli.CliArgsException: Could not build the program from JAR file. at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:203) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968) at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968) Caused by: java.io.FileNotFoundException: JAR file does not exist: -ynm at org.apache.flink.client.cli.CliFrontend.getJarFile(CliFrontend.java:719) at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:695) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:200) ... 4 more Can you please help. Thanks, Anuj On Mon, Apr 13, 2020 at 7:43 AM Yang Wang <[hidden email]> wrote:
|
Hi Anuj, I think the exception you come across still because the hadoop version is 2.4.1. I have checked the hadoop code, the code line are exactly same. For 2.8.1, i also have checked the ruleParse. It could work. /** Could you share the jobmanager logs so that i could check the classpath and hadoop version? Best, Yang aj <[hidden email]> 于2020年4月28日周二 上午1:01写道:
|
Hi Yang, I am attaching the logs for your reference, please help me what i am doing wrong. Thanks, Anuj On Wed, Apr 29, 2020 at 9:06 AM Yang Wang <[hidden email]> wrote:
logs (71K) Download Attachment |
Hello, Please help me upgrade to 1.10 in AWS EMR. On Fri, May 1, 2020 at 4:05 PM aj <[hidden email]> wrote:
|
Hi aj, From the logs you have provided, the hadoop version is still 2.4.1. Could you check the user jar(i.e. events-processor-1.0-SNAPSHOT.jar) have some hadoop classes? If it is, you need to exclude the hadoop dependency. Best, Yang aj <[hidden email]> 于2020年5月6日周三 下午3:38写道:
|
Hello Yang, I have attached my pom file and I did not see that I am using any Hadoop dependency. Can you please help me. On Wed, May 6, 2020 at 1:22 PM Yang Wang <[hidden email]> wrote:
pom.xml (13K) Download Attachment |
Hi Yang, I am able to resolve the issue by removing Hadoop dependency as you mentioned. 1. Removed hadoop-common dependency and <groupId>org.apache.flink</groupId> 2. After the above changes, I am able to submit job on yarn but facing issues with s3 plugin that I resolved by putting flink-s3-fs-hadoop-1.9.0.jar i n the plugins/s3-fs-hadoop directory. Thanks for your support. Any update when will flink.10 officially supported in EMR. Even in new EMR version(emr 6.0) flink has been removed. On Sat, May 9, 2020 at 1:36 PM aj <[hidden email]> wrote:
|
Flink 1.11 will support Hadoop 3. EMR 6 requires Hadoop 3, that's why Flink was not included anymore. Amazon will add Flink back to EMR 6.0 soon. On Thu, May 14, 2020 at 7:11 PM aj <[hidden email]> wrote:
|
Free forum by Nabble | Edit this page |