AWS EMR deployment error : NoClassDefFoundError org/apache/flink/api/java/typeutils/ResultTypeQueryable

[Manas Kale]

Hi,

I am trying to deploy a Flink jar on AWS EMR service. I have ensured that Flink v1.10.0 is used in my pom file as that's the version supported by EMR. However, I get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/typeutils/ResultTypeQueryable
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.typeutils.ResultTypeQueryable
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 15 more

Also, if I deploy this on my local Flink cluster (v1.10.1) it works.

I'm not sure what could be the cause. Could it be because of misconfigured classes bundled in the final JAR file or something that was patched in v 1.10.1?

[Chesnay Schepler]

If this class cannot be found on the classpath then chances are Flink is completely missing from the classpath.

I haven't worked with EMR, but my guess is that you did not submit things correctly.

From the EMR documentation I could gather that the submission should work without the submitted jar bundling all of Flink;

given that you jar works in a local cluster that part should not be the problem.

On 21/08/2020 08:16, Manas Kale wrote:

Hi,

I am trying to deploy a Flink jar on AWS EMR service. I have ensured that Flink v1.10.0 is used in my pom file as that's the version supported by EMR. However, I get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/typeutils/ResultTypeQueryable
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.typeutils.ResultTypeQueryable
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 15 more

Also, if I deploy this on my local Flink cluster (v1.10.1) it works.

I'm not sure what could be the cause. Could it be because of misconfigured classes bundled in the final JAR file or something that was patched in v 1.10.1?

[Prasanna kumar]

Manas, 

One option you could try is to set the scope in the dependencies as <scope>compile</scope> for the required artifacts rather than provided. 

Prasanna.

On Fri, Aug 21, 2020 at 1:47 PM Chesnay Schepler <[hidden email]> wrote:

If this class cannot be found on the classpath then chances are Flink is completely missing from the classpath.

I haven't worked with EMR, but my guess is that you did not submit things correctly.

From the EMR documentation I could gather that the submission should work without the submitted jar bundling all of Flink;

given that you jar works in a local cluster that part should not be the problem.

On 21/08/2020 08:16, Manas Kale wrote:

Hi,

I am trying to deploy a Flink jar on AWS EMR service. I have ensured that Flink v1.10.0 is used in my pom file as that's the version supported by EMR. However, I get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/typeutils/ResultTypeQueryable
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.typeutils.ResultTypeQueryable
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 15 more

Also, if I deploy this on my local Flink cluster (v1.10.1) it works.

I'm not sure what could be the cause. Could it be because of misconfigured classes bundled in the final JAR file or something that was patched in v 1.10.1?

[Manas Kale]

Thanks Prasanna and Chesnay. Changing the dependency scope worked and I also had to add a maven shaded plugin transformer to resolve another error.

On Fri, Aug 21, 2020 at 11:38 PM Prasanna kumar <[hidden email]> wrote:

Manas, 

One option you could try is to set the scope in the dependencies as <scope>compile</scope> for the required artifacts rather than provided. 

Prasanna.

On Fri, Aug 21, 2020 at 1:47 PM Chesnay Schepler <[hidden email]> wrote:

If this class cannot be found on the classpath then chances are Flink is completely missing from the classpath.

I haven't worked with EMR, but my guess is that you did not submit things correctly.

From the EMR documentation I could gather that the submission should work without the submitted jar bundling all of Flink;

given that you jar works in a local cluster that part should not be the problem.

On 21/08/2020 08:16, Manas Kale wrote:

Hi,

I am trying to deploy a Flink jar on AWS EMR service. I have ensured that Flink v1.10.0 is used in my pom file as that's the version supported by EMR. However, I get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/typeutils/ResultTypeQueryable
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.typeutils.ResultTypeQueryable
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 15 more

Also, if I deploy this on my local Flink cluster (v1.10.1) it works.

I'm not sure what could be the cause. Could it be because of misconfigured classes bundled in the final JAR file or something that was patched in v 1.10.1?