Classloader issue using AvroParquetInputFormat via HadoopInputFormat

[Shannon Carey]

Hi folks, congrats on 1.1.0!

FYI, after updating to Flink 1.1.0 I get the exception at bottom when attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink HadoopInputFormat. The ContextUtil.java:71 is trying to execute:

Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");

I am using Scala 2.11.7. JobContextImpl is coming from flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is actually being loaded (according to output with JVM param "-verbose:class") from the flink-shaded-hadoop1_2.10 jar.

After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears to be resolved. Is that the right way to fix the problem?

From what I can tell, the problem is that the JARs that are deployed to Maven Central were built with different versions of Hadoop (as controlled by hadoop.profile):

flink-runtime_2.11 depends on Hadoop 2

flink-java depends on Hadoop 1 (Scala 2.10)

flink-core depends on Hadoop 1 (Scala 2.10)

This seems like a problem with Flink's build process.

As an aside: would it be possible to change the interface of HadoopInputFormat to take a Configuration instead of a Job? That would reduce the dependence on the Hadoop API somewhat. It doesn't look like the Job itself is ever actually used for anything. I'm glad to see you already have https://issues.apache.org/jira/browse/FLINK-4316 and https://issues.apache.org/jira/browse/FLINK-4315 

Thanks,

Shannon

java.lang.IncompatibleClassChangeError: Implementing class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:760)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at org.apache.parquet.hadoop.util.ContextUtil.<clinit>(ContextUtil.java:71)

at org.apache.parquet.avro.AvroParquetInputFormat.setRequestedProjection(AvroParquetInputFormat.java:54)

at com.expedia.www.sdk.flink.HistoricalDataIngestionJob.readHistoricalParquetFile(HistoricalDataIngestionJob.scala:63)

at com.expedia.www.flink.jobs.DestinationCountsHistoryJob$.main(DestinationCountsHistoryJob.scala:25)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply$mcV$sp(DestinationCountsHistoryTest.scala:23)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)

at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)

at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)

at org.scalatest.Transformer.apply(Transformer.scala:22)

at org.scalatest.Transformer.apply(Transformer.scala:20)

at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)

at org.scalatest.Suite$class.withFixture(Suite.scala:1122)

at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)

at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)

at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)

at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)

at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)

at org.scalatest.Suite$class.run(Suite.scala:1424)

at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.SuperEngine.runImpl(Engine.scala:545)

at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.org$scalatest$BeforeAndAfterAll$$super$run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)

at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)

at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)

at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)

at org.scalatest.tools.Runner$.run(Runner.scala:883)

at org.scalatest.tools.Runner.run(Runner.scala)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

[Shannon Carey]

Correction: I cannot work around the problem. If I exclude hadoop1, I get the following exception which appears to be due to flink-java-1.1.0's dependency on Hadoop1.

Failed to submit job 4b6366d101877d38ef33454acc6ca500 (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)

org.apache.flink.runtime.client.JobExecutionException: Failed to submit job 4b6366d101877d38ef33454acc6ca500 (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)

at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281)

at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)

at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)

at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)

at akka.actor.Actor$class.aroundReceive(Actor.scala:465)

at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

at akka.actor.ActorCell.invoke(ActorCell.scala:487)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)

at akka.dispatch.Mailbox.run(Mailbox.scala:221)

at akka.dispatch.Mailbox.exec(Mailbox.scala:231)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: org.apache.flink.runtime.JobException: Creating the input splits caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)

at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695)

at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178)

... 19 more

Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158)

at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56)

at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)

... 21 more

And if I exclude hadoop2, I get the exception from my previous email with AvroParquetInputFormat.

From: Shannon Carey <[hidden email]>
Date: Monday, August 8, 2016 at 2:46 PM
To: "[hidden email]" <[hidden email]>
Subject: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

Hi folks, congrats on 1.1.0!

FYI, after updating to Flink 1.1.0 I get the exception at bottom when attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink HadoopInputFormat. The ContextUtil.java:71 is trying to execute:

Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");

I am using Scala 2.11.7. JobContextImpl is coming from flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is actually being loaded (according to output with JVM param "-verbose:class") from the flink-shaded-hadoop1_2.10 jar.

After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears to be resolved. Is that the right way to fix the problem?

From what I can tell, the problem is that the JARs that are deployed to Maven Central were built with different versions of Hadoop (as controlled by hadoop.profile):

flink-runtime_2.11 depends on Hadoop 2

flink-java depends on Hadoop 1 (Scala 2.10)

flink-core depends on Hadoop 1 (Scala 2.10)

This seems like a problem with Flink's build process.

As an aside: would it be possible to change the interface of HadoopInputFormat to take a Configuration instead of a Job? That would reduce the dependence on the Hadoop API somewhat. It doesn't look like the Job itself is ever actually used for anything. I'm glad to see you already have https://issues.apache.org/jira/browse/FLINK-4316 and https://issues.apache.org/jira/browse/FLINK-4315 

Thanks,

Shannon

java.lang.IncompatibleClassChangeError: Implementing class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:760)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at org.apache.parquet.hadoop.util.ContextUtil.<clinit>(ContextUtil.java:71)

at org.apache.parquet.avro.AvroParquetInputFormat.setRequestedProjection(AvroParquetInputFormat.java:54)

at com.expedia.www.sdk.flink.HistoricalDataIngestionJob.readHistoricalParquetFile(HistoricalDataIngestionJob.scala:63)

at com.expedia.www.flink.jobs.DestinationCountsHistoryJob$.main(DestinationCountsHistoryJob.scala:25)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply$mcV$sp(DestinationCountsHistoryTest.scala:23)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)

at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)

at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)

at org.scalatest.Transformer.apply(Transformer.scala:22)

at org.scalatest.Transformer.apply(Transformer.scala:20)

at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)

at org.scalatest.Suite$class.withFixture(Suite.scala:1122)

at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)

at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)

at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)

at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)

at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)

at org.scalatest.Suite$class.run(Suite.scala:1424)

at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.SuperEngine.runImpl(Engine.scala:545)

at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.org$scalatest$BeforeAndAfterAll$$super$run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)

at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)

at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)

at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)

at org.scalatest.tools.Runner$.run(Runner.scala:883)

at org.scalatest.tools.Runner.run(Runner.scala)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

[Stephan Ewen]

Hi Shannon!

It seams that the something in the maven deployment went wrong with this release.

There should be:

  - flink-java (the default, with a transitive dependency to hadoop 2.x for hadoop compatibility features)

  - flink-java-hadoop1 (with a transitive dependency for hadoop 1.x fir older hadoop compatibility features)

Apparently the "flink-java" artifact git overwritten with the "flink-java-hadoop1" artifact. Damn.

I think we need to release new artifacts that fix these dependency descriptors.

That needs to be a 1.1.1 release, because maven artifacts cannot be changed after they were deployed.

Greetings,
Stephan

On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey <[hidden email]> wrote:

Correction: I cannot work around the problem. If I exclude hadoop1, I get the following exception which appears to be due to flink-java-1.1.0's dependency on Hadoop1.

Failed to submit job 4b6366d101877d38ef33454acc6ca500 (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)

org.apache.flink.runtime.client.JobExecutionException: Failed to submit job 4b6366d101877d38ef33454acc6ca500 (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)

at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281)

at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)

at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)

at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)

at akka.actor.Actor$class.aroundReceive(Actor.scala:465)

at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

at akka.actor.ActorCell.invoke(ActorCell.scala:487)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)

at akka.dispatch.Mailbox.run(Mailbox.scala:221)

at akka.dispatch.Mailbox.exec(Mailbox.scala:231)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: org.apache.flink.runtime.JobException: Creating the input splits caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)

at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695)

at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178)

... 19 more

Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158)

at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56)

at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)

... 21 more

And if I exclude hadoop2, I get the exception from my previous email with AvroParquetInputFormat.

From: Shannon Carey <[hidden email]>
Date: Monday, August 8, 2016 at 2:46 PM
To: "[hidden email]" <[hidden email]>
Subject: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

Hi folks, congrats on 1.1.0!

FYI, after updating to Flink 1.1.0 I get the exception at bottom when attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink HadoopInputFormat. The ContextUtil.java:71 is trying to execute:

Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");

I am using Scala 2.11.7. JobContextImpl is coming from flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is actually being loaded (according to output with JVM param "-verbose:class") from the flink-shaded-hadoop1_2.10 jar.

After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears to be resolved. Is that the right way to fix the problem?

From what I can tell, the problem is that the JARs that are deployed to Maven Central were built with different versions of Hadoop (as controlled by hadoop.profile):

flink-runtime_2.11 depends on Hadoop 2

flink-java depends on Hadoop 1 (Scala 2.10)

flink-core depends on Hadoop 1 (Scala 2.10)

This seems like a problem with Flink's build process.

As an aside: would it be possible to change the interface of HadoopInputFormat to take a Configuration instead of a Job? That would reduce the dependence on the Hadoop API somewhat. It doesn't look like the Job itself is ever actually used for anything. I'm glad to see you already have https://issues.apache.org/jira/browse/FLINK-4316 and https://issues.apache.org/jira/browse/FLINK-4315 

Thanks,

Shannon

java.lang.IncompatibleClassChangeError: Implementing class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:760)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at org.apache.parquet.hadoop.util.ContextUtil.<clinit>(ContextUtil.java:71)

at org.apache.parquet.avro.AvroParquetInputFormat.setRequestedProjection(AvroParquetInputFormat.java:54)

at com.expedia.www.sdk.flink.HistoricalDataIngestionJob.readHistoricalParquetFile(HistoricalDataIngestionJob.scala:63)

at com.expedia.www.flink.jobs.DestinationCountsHistoryJob$.main(DestinationCountsHistoryJob.scala:25)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply$mcV$sp(DestinationCountsHistoryTest.scala:23)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)

at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)

at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)

at org.scalatest.Transformer.apply(Transformer.scala:22)

at org.scalatest.Transformer.apply(Transformer.scala:20)

at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:<a href="tel:1647" value="+491647" target="_blank">1647)

at org.scalatest.Suite$class.withFixture(Suite.scala:1122)

at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:<a href="tel:1683" value="+491683" target="_blank">1683)

at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:<a href="tel:1644" value="+491644" target="_blank">1644)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)

at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpec.runTest(FlatSpec.scala:<a href="tel:1683" value="+491683" target="_blank">1683)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)

at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)

at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpec.runTests(FlatSpec.scala:<a href="tel:1683" value="+491683" target="_blank">1683)

at org.scalatest.Suite$class.run(Suite.scala:1424)

at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:<a href="tel:1683" value="+491683" target="_blank">1683)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.SuperEngine.runImpl(Engine.scala:545)

at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.org$scalatest$BeforeAndAfterAll$$super$run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)

at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)

at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)

at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)

at org.scalatest.tools.Runner$.run(Runner.scala:883)

at org.scalatest.tools.Runner.run(Runner.scala)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

[LINZ, Arnaud]

Okay,

That would also solve my issue.

Greetings,

Arnaud

 

De : Stephan Ewen [mailto:[hidden email]]
Envoyé : mardi 9 août 2016 12:41
À : [hidden email]
Objet : Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

 

Hi Shannon!

 

It seams that the something in the maven deployment went wrong with this release.

 

There should be:

  - flink-java (the default, with a transitive dependency to hadoop 2.x for hadoop compatibility features)

  - flink-java-hadoop1 (with a transitive dependency for hadoop 1.x fir older hadoop compatibility features)

 

Apparently the "flink-java" artifact git overwritten with the "flink-java-hadoop1" artifact. Damn.

 

I think we need to release new artifacts that fix these dependency descriptors.

 

That needs to be a 1.1.1 release, because maven artifacts cannot be changed after they were deployed.

 

Greetings,
Stephan

 

 

 

 

 

 

On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey <[hidden email]> wrote:

Correction: I cannot work around the problem. If I exclude hadoop1, I get the following exception which appears to be due to flink-java-1.1.0's dependency on Hadoop1.

 

Failed to submit job 4b6366d101877d38ef33454acc6ca500 (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)

org.apache.flink.runtime.client.JobExecutionException: Failed to submit job 4b6366d101877d38ef33454acc6ca500 (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)

at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281)

at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)

at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)

at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)

at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)

at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)

at akka.actor.Actor$class.aroundReceive(Actor.scala:465)

at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121)

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

at akka.actor.ActorCell.invoke(ActorCell.scala:487)

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)

at akka.dispatch.Mailbox.run(Mailbox.scala:221)

at akka.dispatch.Mailbox.exec(Mailbox.scala:231)

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: org.apache.flink.runtime.JobException: Creating the input splits caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)

at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695)

at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178)

... 19 more

Caused by: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158)

at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56)

at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)

... 21 more

 

And if I exclude hadoop2, I get the exception from my previous email with AvroParquetInputFormat.

 

 

 

From: Shannon Carey <[hidden email]>
Date: Monday, August 8, 2016 at 2:46 PM
To: "[hidden email]" <[hidden email]>
Subject: Classloader issue using AvroParquetInputFormat via HadoopInputFormat

 

Hi folks, congrats on 1.1.0!

 

FYI, after updating to Flink 1.1.0 I get the exception at bottom when attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink HadoopInputFormat. The ContextUtil.java:71 is trying to execute:

 

Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");

 

I am using Scala 2.11.7. JobContextImpl is coming from flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is actually being loaded (according to output with JVM param "-verbose:class") from the flink-shaded-hadoop1_2.10 jar.

 

After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears to be resolved. Is that the right way to fix the problem?

 

From what I can tell, the problem is that the JARs that are deployed to Maven Central were built with different versions of Hadoop (as controlled by hadoop.profile):

 

flink-runtime_2.11 depends on Hadoop 2

flink-java depends on Hadoop 1 (Scala 2.10)

flink-core depends on Hadoop 1 (Scala 2.10)

 

This seems like a problem with Flink's build process.

 

As an aside: would it be possible to change the interface of HadoopInputFormat to take a Configuration instead of a Job? That would reduce the dependence on the Hadoop API somewhat. It doesn't look like the Job itself is ever actually used for anything. I'm glad to see you already have https://issues.apache.org/jira/browse/FLINK-4316 and https://issues.apache.org/jira/browse/FLINK-4315 

 

Thanks,

Shannon

 

java.lang.IncompatibleClassChangeError: Implementing class

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:760)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at org.apache.parquet.hadoop.util.ContextUtil.<clinit>(ContextUtil.java:71)

at org.apache.parquet.avro.AvroParquetInputFormat.setRequestedProjection(AvroParquetInputFormat.java:54)

at com.expedia.www.sdk.flink.HistoricalDataIngestionJob.readHistoricalParquetFile(HistoricalDataIngestionJob.scala:63)

at com.expedia.www.flink.jobs.DestinationCountsHistoryJob$.main(DestinationCountsHistoryJob.scala:25)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply$mcV$sp(DestinationCountsHistoryTest.scala:23)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)

at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)

at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)

at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)

at org.scalatest.Transformer.apply(Transformer.scala:22)

at org.scalatest.Transformer.apply(Transformer.scala:20)

at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:<a href="tel:1647" target="_blank">1647)

at org.scalatest.Suite$class.withFixture(Suite.scala:1122)

at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:<a href="tel:1683" target="_blank">1683)

at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:<a href="tel:1644" target="_blank">1644)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)

at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)

at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)

at org.scalatest.FlatSpec.runTest(FlatSpec.scala:<a href="tel:1683" target="_blank">1683)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)

at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)

at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)

at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)

at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)

at org.scalatest.FlatSpec.runTests(FlatSpec.scala:<a href="tel:1683" target="_blank">1683)

at org.scalatest.Suite$class.run(Suite.scala:1424)

at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:<a href="tel:1683" target="_blank">1683)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)

at org.scalatest.SuperEngine.runImpl(Engine.scala:545)

at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.org$scalatest$BeforeAndAfterAll$$super$run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)

at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)

at com.expedia.www.flink.jobs.DestinationCountsHistoryTest.run(DestinationCountsHistoryTest.scala:12)

at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)

at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)

at scala.collection.immutable.List.foreach(List.scala:381)

at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)

at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)

at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)

at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)

at org.scalatest.tools.Runner$.run(Runner.scala:883)

at org.scalatest.tools.Runner.run(Runner.scala)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)

at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

 

 

 

---

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

[Ufuk Celebi]

This is a problem with the Maven artifacts of 1.1.0 :-( I've added a
warning to the release note and will start a emergency vote for 1.1.1
which only updates the Maven artifacts.

On Tue, Aug 9, 2016 at 1:20 PM, LINZ, Arnaud <[hidden email]> wrote:

> Okay,
>
> That would also solve my issue.
>
> Greetings,
>
> Arnaud
>
>
>
> De : Stephan Ewen [mailto:[hidden email]]
> Envoyé : mardi 9 août 2016 12:41
> À : [hidden email]
> Objet : Re: Classloader issue using AvroParquetInputFormat via
> HadoopInputFormat
>
>
>
> Hi Shannon!
>
>
>
> It seams that the something in the maven deployment went wrong with this
> release.
>
>
>
> There should be:
>
>   - flink-java (the default, with a transitive dependency to hadoop 2.x for
> hadoop compatibility features)
>
>   - flink-java-hadoop1 (with a transitive dependency for hadoop 1.x fir
> older hadoop compatibility features)
>
>
>
> Apparently the "flink-java" artifact git overwritten with the
> "flink-java-hadoop1" artifact. Damn.
>
>
>
> I think we need to release new artifacts that fix these dependency
> descriptors.
>
>
>
> That needs to be a 1.1.1 release, because maven artifacts cannot be changed
> after they were deployed.
>
>
>
> Greetings,
> Stephan
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey <[hidden email]> wrote:
>
> Correction: I cannot work around the problem. If I exclude hadoop1, I get
> the following exception which appears to be due to flink-java-1.1.0's
> dependency on Hadoop1.
>
>
>
> Failed to submit job 4b6366d101877d38ef33454acc6ca500
> (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)
>
> org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
> 4b6366d101877d38ef33454acc6ca500
> (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)
>
> at
> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281)
>
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)
>
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>
> at
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
>
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>
> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>
> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>
> at
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
> at
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121)
>
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> Caused by: org.apache.flink.runtime.JobException: Creating the input splits
> caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but
> class was expected
>
> at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)
>
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695)
>
> at
> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178)
>
> ... 19 more
>
> Caused by: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>
> at
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158)
>
> at
> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56)
>
> at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
>
> ... 21 more
>
>
>
> And if I exclude hadoop2, I get the exception from my previous email with
> AvroParquetInputFormat.
>
>
>
>
>
>
>
> From: Shannon Carey <[hidden email]>
> Date: Monday, August 8, 2016 at 2:46 PM
> To: "[hidden email]" <[hidden email]>
> Subject: Classloader issue using AvroParquetInputFormat via
> HadoopInputFormat
>
>
>
> Hi folks, congrats on 1.1.0!
>
>
>
> FYI, after updating to Flink 1.1.0 I get the exception at bottom when
> attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink
> HadoopInputFormat. The ContextUtil.java:71 is trying to execute:
>
>
>
> Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");
>
>
>
> I am using Scala 2.11.7. JobContextImpl is coming from
> flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is
> actually being loaded (according to output with JVM param "-verbose:class")
> from the flink-shaded-hadoop1_2.10 jar.
>
>
>
> After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears
> to be resolved. Is that the right way to fix the problem?
>
>
>
> From what I can tell, the problem is that the JARs that are deployed to
> Maven Central were built with different versions of Hadoop (as controlled by
> hadoop.profile):
>
>
>
> flink-runtime_2.11 depends on Hadoop 2
>
> flink-java depends on Hadoop 1 (Scala 2.10)
>
> flink-core depends on Hadoop 1 (Scala 2.10)
>
>
>
> This seems like a problem with Flink's build process.
>
>
>
> As an aside: would it be possible to change the interface of
> HadoopInputFormat to take a Configuration instead of a Job? That would
> reduce the dependence on the Hadoop API somewhat. It doesn't look like the
> Job itself is ever actually used for anything. I'm glad to see you already
> have https://issues.apache.org/jira/browse/FLINK-4316 and
> https://issues.apache.org/jira/browse/FLINK-4315
>
>
>
> Thanks,
>
> Shannon
>
>
>
> java.lang.IncompatibleClassChangeError: Implementing class
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:264)
>
> at org.apache.parquet.hadoop.util.ContextUtil.<clinit>(ContextUtil.java:71)
>
> at
> org.apache.parquet.avro.AvroParquetInputFormat.setRequestedProjection(AvroParquetInputFormat.java:54)
>
> at
> com.expedia.www.sdk.flink.HistoricalDataIngestionJob.readHistoricalParquetFile(HistoricalDataIngestionJob.scala:63)
>
> at
> com.expedia.www.flink.jobs.DestinationCountsHistoryJob$.main(DestinationCountsHistoryJob.scala:25)
>
> at
> com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply$mcV$sp(DestinationCountsHistoryTest.scala:23)
>
> at
> com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)
>
> at
> com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)
>
> at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>
> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>
> at org.scalatest.Transformer.apply(Transformer.scala:22)
>
> at org.scalatest.Transformer.apply(Transformer.scala:20)
>
> at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
>
> at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
>
> at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
>
> at
> org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644)
>
> at
> org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
>
> at
> org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
>
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>
> at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)
>
> at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)
>
> at
> org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
>
> at
> org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
>
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>
> at scala.collection.immutable.List.foreach(List.scala:381)
>
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>
> at
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)
>
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)
>
> at
> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>
> at scala.collection.immutable.List.foreach(List.scala:381)
>
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>
> at
> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>
> at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)
>
> at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)
>
> at org.scalatest.Suite$class.run(Suite.scala:1424)
>
> at
> org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)
>
> at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
>
> at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
>
> at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>
> at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)
>
> at
> com.expedia.www.flink.jobs.DestinationCountsHistoryTest.org$scalatest$BeforeAndAfterAll$$super$run(DestinationCountsHistoryTest.scala:12)
>
> at
> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>
> at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>
> at
> com.expedia.www.flink.jobs.DestinationCountsHistoryTest.run(DestinationCountsHistoryTest.scala:12)
>
> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
>
> at
> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
>
> at
> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
>
> at scala.collection.immutable.List.foreach(List.scala:381)
>
> at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
>
> at
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
>
> at
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
>
> at
> org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
>
> at
> org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
>
> at org.scalatest.tools.Runner$.run(Runner.scala:883)
>
> at org.scalatest.tools.Runner.run(Runner.scala)
>
> at
> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)
>
> at
> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
>
>
>
>
>
>
>
>
> ________________________________
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous
> n'êtes pas destinataire de ce message, merci de le détruire et d'avertir
> l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.

[Ufuk Celebi]

I've started a vote for 1.1.1 containing hopefully fixed artifacts. If
you have any spare time, would you mind checking whether it fixes your
problem?

The artifacts are here: http://home.apache.org/~uce/flink-1.1.1-rc1/

You would have to add the following repository to your Maven project
and update the Flink version to 1.1.1:

<repositories>
<repository>
<id>flink-rc</id>
<name>flink-rc</name>
<url>https://repository.apache.org/content/repositories/orgapacheflink-1101</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>

Would really appreciate it!

On Tue, Aug 9, 2016 at 2:11 PM, Ufuk Celebi <[hidden email]> wrote:

> This is a problem with the Maven artifacts of 1.1.0 :-( I've added a
> warning to the release note and will start a emergency vote for 1.1.1
> which only updates the Maven artifacts.
>
> On Tue, Aug 9, 2016 at 1:20 PM, LINZ, Arnaud <[hidden email]> wrote:
>> Okay,
>>
>> That would also solve my issue.
>>
>> Greetings,
>>
>> Arnaud
>>
>>
>>
>> De : Stephan Ewen [mailto:[hidden email]]
>> Envoyé : mardi 9 août 2016 12:41
>> À : [hidden email]
>> Objet : Re: Classloader issue using AvroParquetInputFormat via
>> HadoopInputFormat
>>
>>
>>
>> Hi Shannon!
>>
>>
>>
>> It seams that the something in the maven deployment went wrong with this
>> release.
>>
>>
>>
>> There should be:
>>
>>   - flink-java (the default, with a transitive dependency to hadoop 2.x for
>> hadoop compatibility features)
>>
>>   - flink-java-hadoop1 (with a transitive dependency for hadoop 1.x fir
>> older hadoop compatibility features)
>>
>>
>>
>> Apparently the "flink-java" artifact git overwritten with the
>> "flink-java-hadoop1" artifact. Damn.
>>
>>
>>
>> I think we need to release new artifacts that fix these dependency
>> descriptors.
>>
>>
>>
>> That needs to be a 1.1.1 release, because maven artifacts cannot be changed
>> after they were deployed.
>>
>>
>>
>> Greetings,
>> Stephan
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey <[hidden email]> wrote:
>>
>> Correction: I cannot work around the problem. If I exclude hadoop1, I get
>> the following exception which appears to be due to flink-java-1.1.0's
>> dependency on Hadoop1.
>>
>>
>>
>> Failed to submit job 4b6366d101877d38ef33454acc6ca500
>> (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)
>>
>> org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
>> 4b6366d101877d38ef33454acc6ca500
>> (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)
>>
>> at
>> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281)
>>
>> at
>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)
>>
>> at
>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>
>> at
>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
>>
>> at
>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>
>> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>>
>> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>>
>> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>
>> at
>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>>
>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>
>> at
>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121)
>>
>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>>
>> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>>
>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>
>> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>
>> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>
>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>
>> at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>
>> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> Caused by: org.apache.flink.runtime.JobException: Creating the input splits
>> caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but
>> class was expected
>>
>> at
>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)
>>
>> at
>> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695)
>>
>> at
>> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178)
>>
>> ... 19 more
>>
>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>
>> at
>> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158)
>>
>> at
>> org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56)
>>
>> at
>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
>>
>> ... 21 more
>>
>>
>>
>> And if I exclude hadoop2, I get the exception from my previous email with
>> AvroParquetInputFormat.
>>
>>
>>
>>
>>
>>
>>
>> From: Shannon Carey <[hidden email]>
>> Date: Monday, August 8, 2016 at 2:46 PM
>> To: "[hidden email]" <[hidden email]>
>> Subject: Classloader issue using AvroParquetInputFormat via
>> HadoopInputFormat
>>
>>
>>
>> Hi folks, congrats on 1.1.0!
>>
>>
>>
>> FYI, after updating to Flink 1.1.0 I get the exception at bottom when
>> attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink
>> HadoopInputFormat. The ContextUtil.java:71 is trying to execute:
>>
>>
>>
>> Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");
>>
>>
>>
>> I am using Scala 2.11.7. JobContextImpl is coming from
>> flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is
>> actually being loaded (according to output with JVM param "-verbose:class")
>> from the flink-shaded-hadoop1_2.10 jar.
>>
>>
>>
>> After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears
>> to be resolved. Is that the right way to fix the problem?
>>
>>
>>
>> From what I can tell, the problem is that the JARs that are deployed to
>> Maven Central were built with different versions of Hadoop (as controlled by
>> hadoop.profile):
>>
>>
>>
>> flink-runtime_2.11 depends on Hadoop 2
>>
>> flink-java depends on Hadoop 1 (Scala 2.10)
>>
>> flink-core depends on Hadoop 1 (Scala 2.10)
>>
>>
>>
>> This seems like a problem with Flink's build process.
>>
>>
>>
>> As an aside: would it be possible to change the interface of
>> HadoopInputFormat to take a Configuration instead of a Job? That would
>> reduce the dependence on the Hadoop API somewhat. It doesn't look like the
>> Job itself is ever actually used for anything. I'm glad to see you already
>> have https://issues.apache.org/jira/browse/FLINK-4316 and
>> https://issues.apache.org/jira/browse/FLINK-4315
>>
>>
>>
>> Thanks,
>>
>> Shannon
>>
>>
>>
>> java.lang.IncompatibleClassChangeError: Implementing class
>>
>> at java.lang.ClassLoader.defineClass1(Native Method)
>>
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
>>
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>
>> at java.lang.Class.forName0(Native Method)
>>
>> at java.lang.Class.forName(Class.java:264)
>>
>> at org.apache.parquet.hadoop.util.ContextUtil.<clinit>(ContextUtil.java:71)
>>
>> at
>> org.apache.parquet.avro.AvroParquetInputFormat.setRequestedProjection(AvroParquetInputFormat.java:54)
>>
>> at
>> com.expedia.www.sdk.flink.HistoricalDataIngestionJob.readHistoricalParquetFile(HistoricalDataIngestionJob.scala:63)
>>
>> at
>> com.expedia.www.flink.jobs.DestinationCountsHistoryJob$.main(DestinationCountsHistoryJob.scala:25)
>>
>> at
>> com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply$mcV$sp(DestinationCountsHistoryTest.scala:23)
>>
>> at
>> com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)
>>
>> at
>> com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)
>>
>> at
>> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
>>
>> at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
>>
>> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
>>
>> at org.scalatest.Transformer.apply(Transformer.scala:22)
>>
>> at org.scalatest.Transformer.apply(Transformer.scala:20)
>>
>> at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
>>
>> at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
>>
>> at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
>>
>> at
>> org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644)
>>
>> at
>> org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
>>
>> at
>> org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
>>
>> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
>>
>> at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)
>>
>> at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)
>>
>> at
>> org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
>>
>> at
>> org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
>>
>> at
>> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
>>
>> at
>> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>>
>> at scala.collection.immutable.List.foreach(List.scala:381)
>>
>> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>>
>> at
>> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)
>>
>> at
>> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)
>>
>> at
>> org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
>>
>> at scala.collection.immutable.List.foreach(List.scala:381)
>>
>> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
>>
>> at
>> org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
>>
>> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
>>
>> at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)
>>
>> at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)
>>
>> at org.scalatest.Suite$class.run(Suite.scala:1424)
>>
>> at
>> org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)
>>
>> at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
>>
>> at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
>>
>> at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
>>
>> at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)
>>
>> at
>> com.expedia.www.flink.jobs.DestinationCountsHistoryTest.org$scalatest$BeforeAndAfterAll$$super$run(DestinationCountsHistoryTest.scala:12)
>>
>> at
>> org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
>>
>> at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
>>
>> at
>> com.expedia.www.flink.jobs.DestinationCountsHistoryTest.run(DestinationCountsHistoryTest.scala:12)
>>
>> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
>>
>> at
>> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
>>
>> at
>> org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
>>
>> at scala.collection.immutable.List.foreach(List.scala:381)
>>
>> at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
>>
>> at
>> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
>>
>> at
>> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
>>
>> at
>> org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
>>
>> at
>> org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
>>
>> at org.scalatest.tools.Runner$.run(Runner.scala:883)
>>
>> at org.scalatest.tools.Runner.run(Runner.scala)
>>
>> at
>> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)
>>
>> at
>> org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
>>
>>
>>
>>
>>
>>
>>
>>
>> ________________________________
>>
>> L'intégrité de ce message n'étant pas assurée sur internet, la société
>> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
>> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous
>> n'êtes pas destinataire de ce message, merci de le détruire et d'avertir
>> l'expéditeur.
>>
>> The integrity of this message cannot be guaranteed on the Internet. The
>> company that sent this message cannot therefore be held liable for its
>> content nor attachments. Any unauthorized use or dissemination is
>> prohibited. If you are not the intended recipient of this message, then
>> please delete it and notify the sender.