(DEPRECATED) Apache Flink User Mailing List archive.

Error after upgrading to Flink 1.6.2

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Flavio Pompermaier
Reply | Threaded
Open this post in threaded view
♦
♦
|

Error after upgrading to Flink 1.6.2

Flavio Pompermaier
Hi to all,
we tried to upgrade our jobs to Flink 1.6.2 but now we get the following error (we saw a similar issue with spark that was caused by different java version on the cluster servers so we checked them and they are all to the same version - oracle-8-191):

Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'DataSink (Parquet write: hdfs:/rivela/1/1/0_staging/parquet)': Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:220)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
	at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1151)
	at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1131)
	at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:294)
	at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
	... 10 more
Caused by: java.lang.Exception: Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:63)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216)
	... 15 more
Caused by: java.lang.IllegalStateException: unread block data
	at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2783)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1605)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:502)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:489)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:477)
	at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:438)
	at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288)
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:60)
	... 16 more

Has anyone faced this problem before? How can we try to solve it?
Best,Flavio
Flavio Pompermaier
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Error after upgrading to Flink 1.6.2

Flavio Pompermaier
I forgot to mention that I'm using Flink 1.6.2 compiled for cloudera CDH 5.11.2:

/opt/shared/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.11.2 -Dhbase.version=1.2.0-cdh5.11.2 -Dhadoop.core.version=2.6.0-mr1-cdh5.11.2 -DskipTests -Pvendor-repos

On Wed, Nov 7, 2018 at 11:48 AM Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
we tried to upgrade our jobs to Flink 1.6.2 but now we get the following error (we saw a similar issue with spark that was caused by different java version on the cluster servers so we checked them and they are all to the same version - oracle-8-191):

Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'DataSink (Parquet write: hdfs:/rivela/1/1/0_staging/parquet)': Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:220)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
	at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1151)
	at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1131)
	at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:294)
	at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
	... 10 more
Caused by: java.lang.Exception: Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:63)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216)
	... 15 more
Caused by: java.lang.IllegalStateException: unread block data
	at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2783)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1605)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:502)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:489)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:477)
	at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:438)
	at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288)
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:60)
	... 16 more

Has anyone faced this problem before? How can we try to solve it?
Best,Flavio
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Error after upgrading to Flink 1.6.2

Till Rohrmann
Hi Flavio,

I haven't seen this problem before. Are you using Flink's HBase connector? According to similar problems with Spark one needs to make sure that the hbase jars are on the classpath [1, 2]. If not, then it might be a problem with the MR1 version 2.6.0-mr1-cdh5.11.2 which caused problems for CDH 5.2 [2]. It could also be worthwhile to try it out with the latest CDH version.


Cheers,
Till

On Wed, Nov 7, 2018 at 12:05 PM Flavio Pompermaier <[hidden email]> wrote:
I forgot to mention that I'm using Flink 1.6.2 compiled for cloudera CDH 5.11.2:

/opt/shared/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.11.2 -Dhbase.version=1.2.0-cdh5.11.2 -Dhadoop.core.version=2.6.0-mr1-cdh5.11.2 -DskipTests -Pvendor-repos

On Wed, Nov 7, 2018 at 11:48 AM Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
we tried to upgrade our jobs to Flink 1.6.2 but now we get the following error (we saw a similar issue with spark that was caused by different java version on the cluster servers so we checked them and they are all to the same version - oracle-8-191):

Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'DataSink (Parquet write: hdfs:/rivela/1/1/0_staging/parquet)': Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:220)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
	at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1151)
	at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1131)
	at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:294)
	at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
	... 10 more
Caused by: java.lang.Exception: Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:63)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216)
	... 15 more
Caused by: java.lang.IllegalStateException: unread block data
	at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2783)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1605)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:502)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:489)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:477)
	at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:438)
	at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288)
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:60)
	... 16 more

Has anyone faced this problem before? How can we try to solve it?
Best,Flavio
Flavio Pompermaier
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Error after upgrading to Flink 1.6.2

Flavio Pompermaier
Hi Till,
we are not using HBase at the moment. We managed to run successfully the job but it was a pain to find the right combination of dependencies, library shading and the right HADOOP_CLASSPATH.
The problem was the combination of parquet, jaxrs, hadoop and jackson.
Moreover we had to run the cluster with parent-first class loading in order to make it run. 

However we still have the big problem of being able to submit jobs via rest API (as I wrote in another thread it seems that there's no way to execute any code after env.execute if using REST APIs).

Best,
Flavio
 
On Wed, Nov 7, 2018 at 6:15 PM Till Rohrmann <[hidden email]> wrote:
Hi Flavio,

I haven't seen this problem before. Are you using Flink's HBase connector? According to similar problems with Spark one needs to make sure that the hbase jars are on the classpath [1, 2]. If not, then it might be a problem with the MR1 version 2.6.0-mr1-cdh5.11.2 which caused problems for CDH 5.2 [2]. It could also be worthwhile to try it out with the latest CDH version.


Cheers,
Till

On Wed, Nov 7, 2018 at 12:05 PM Flavio Pompermaier <[hidden email]> wrote:
I forgot to mention that I'm using Flink 1.6.2 compiled for cloudera CDH 5.11.2:

/opt/shared/devel/apache-maven-3.3.9/bin/mvn clean install -Dhadoop.version=2.6.0-cdh5.11.2 -Dhbase.version=1.2.0-cdh5.11.2 -Dhadoop.core.version=2.6.0-mr1-cdh5.11.2 -DskipTests -Pvendor-repos

On Wed, Nov 7, 2018 at 11:48 AM Flavio Pompermaier <[hidden email]> wrote:
Hi to all,
we tried to upgrade our jobs to Flink 1.6.2 but now we get the following error (we saw a similar issue with spark that was caused by different java version on the cluster servers so we checked them and they are all to the same version - oracle-8-191):

Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'DataSink (Parquet write: hdfs:/rivela/1/1/0_staging/parquet)': Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:220)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
	at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1151)
	at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1131)
	at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:294)
	at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
	... 10 more
Caused by: java.lang.Exception: Deserializing the OutputFormat (org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat@54a4c7c8) failed: unread block data
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:63)
	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:216)
	... 15 more
Caused by: java.lang.IllegalStateException: unread block data
	at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2783)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1605)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:502)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:489)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:477)
	at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:438)
	at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288)
	at org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:60)
	... 16 more

Has anyone faced this problem before? How can we try to solve it?
Best,Flavio
Free forum by Nabble Edit this page