(DEPRECATED) Apache Flink User Mailing List archive.

Exception in Simple Job: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method

classic Classic list List threaded Threaded
♦
♦
 4 messages Options
Yiannis Gkoufas
Reply | Threaded
Open this post in threaded view
♦
♦
|

Exception in Simple Job: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method

Yiannis Gkoufas
Hi there,

not sure if its a bug of 0.9-SNAPSHOT version, or me just doing something wrong.
I have this simple scala program:

val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
input.map(e => e.split(",")).filter(e => e.length>=4)
.map(e => (e(1),e(2),e(3).toDouble))
.groupBy(0,1)
.sum(2)
.writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")

env.execute("Test Flink")

And I get the following error:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce (SUM(2))' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
	at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
	at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
	at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
	at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
	... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
Caused by: java.io.IOException: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
Caused by: java.lang.IllegalStateException: The runtime context has not been initialized.
	at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
	at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
	... 1 more

	at org.apache.flink.client.program.Client.run(Client.java:345)
	at org.apache.flink.client.program.Client.run(Client.java:304)
	at org.apache.flink.client.program.Client.run(Client.java:298)
	at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
	at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
	at com.gmp.MyJob$.main(MyJob.scala:33)
	at com.gmp.MyJob.main(MyJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
	at org.apache.flink.client.program.Client.run(Client.java:250)
	at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
Any help would be really appreciated!

Thank you!
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Exception in Simple Job: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method

Fabian Hueske-2
Hi,

you are doing everything correct.
 
This is a bug in the Flink runtime.
I created a JIRA (https://issues.apache.org/jira/browse/FLINK-1574) and will push a fix later this evening once all tests have passed.

Thanks for reporting the issue!

Cheers, Fabian

2015-02-17 18:00 GMT+01:00 Yiannis Gkoufas <[hidden email]>:
Hi there,

not sure if its a bug of 0.9-SNAPSHOT version, or me just doing something wrong.
I have this simple scala program:

val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
input.map(e => e.split(",")).filter(e => e.length>=4)
.map(e => (e(1),e(2),e(3).toDouble))
.groupBy(0,1)
.sum(2)
.writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")

env.execute("Test Flink")

And I get the following error:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce (SUM(2))' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
	at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
	at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
	at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
	at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
	... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
Caused by: java.io.IOException: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
Caused by: java.lang.IllegalStateException: The runtime context has not been initialized.
	at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
	at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
	... 1 more

	at org.apache.flink.client.program.Client.run(Client.java:345)
	at org.apache.flink.client.program.Client.run(Client.java:304)
	at org.apache.flink.client.program.Client.run(Client.java:298)
	at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
	at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
	at com.gmp.MyJob$.main(MyJob.scala:33)
	at com.gmp.MyJob.main(MyJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
	at org.apache.flink.client.program.Client.run(Client.java:250)
	at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
Any help would be really appreciated!

Thank you!
Fabian Hueske-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Exception in Simple Job: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method

Fabian Hueske-2

2015-02-17 18:57 GMT+01:00 Fabian Hueske <[hidden email]>:
Hi,

you are doing everything correct.
 
This is a bug in the Flink runtime.
I created a JIRA (https://issues.apache.org/jira/browse/FLINK-1574) and will push a fix later this evening once all tests have passed.

Thanks for reporting the issue!

Cheers, Fabian

2015-02-17 18:00 GMT+01:00 Yiannis Gkoufas <[hidden email]>:
Hi there,

not sure if its a bug of 0.9-SNAPSHOT version, or me just doing something wrong.
I have this simple scala program:

val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
input.map(e => e.split(",")).filter(e => e.length>=4)
.map(e => (e(1),e(2),e(3).toDouble))
.groupBy(0,1)
.sum(2)
.writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")

env.execute("Test Flink")

And I get the following error:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce (SUM(2))' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
	at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
	at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
	at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
	at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
	... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
Caused by: java.io.IOException: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
Caused by: java.lang.IllegalStateException: The runtime context has not been initialized.
	at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
	at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
	... 1 more

	at org.apache.flink.client.program.Client.run(Client.java:345)
	at org.apache.flink.client.program.Client.run(Client.java:304)
	at org.apache.flink.client.program.Client.run(Client.java:298)
	at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
	at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
	at com.gmp.MyJob$.main(MyJob.scala:33)
	at com.gmp.MyJob.main(MyJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
	at org.apache.flink.client.program.Client.run(Client.java:250)
	at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
Any help would be really appreciated!

Thank you!
Yiannis Gkoufas
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Exception in Simple Job: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method

Yiannis Gkoufas

Hi Fabian,

Thanks for letting me know! I will just give it a shot tomorrow.

Regards

On 17 Feb 2015 18:02, "Fabian Hueske" <[hidden email]> wrote:

2015-02-17 18:57 GMT+01:00 Fabian Hueske <[hidden email]>:
Hi,

you are doing everything correct.
 
This is a bug in the Flink runtime.
I created a JIRA (https://issues.apache.org/jira/browse/FLINK-1574) and will push a fix later this evening once all tests have passed.

Thanks for reporting the issue!

Cheers, Fabian

2015-02-17 18:00 GMT+01:00 Yiannis Gkoufas <[hidden email]>:
Hi there,

not sure if its a bug of 0.9-SNAPSHOT version, or me just doing something wrong.
I have this simple scala program:

val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
input.map(e => e.split(",")).filter(e => e.length>=4)
.map(e => (e(1),e(2),e(3).toDouble))
.groupBy(0,1)
.sum(2)
.writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")

env.execute("Test Flink")

And I get the following error:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce (SUM(2))' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
	at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
	at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
	at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
	at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
	... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
Caused by: java.io.IOException: The user-defined combiner failed in its 'open()' method.
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
Caused by: java.lang.IllegalStateException: The runtime context has not been initialized.
	at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
	at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
	at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
	... 1 more

	at org.apache.flink.client.program.Client.run(Client.java:345)
	at org.apache.flink.client.program.Client.run(Client.java:304)
	at org.apache.flink.client.program.Client.run(Client.java:298)
	at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
	at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
	at com.gmp.MyJob$.main(MyJob.scala:33)
	at com.gmp.MyJob.main(MyJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
	at org.apache.flink.client.program.Client.run(Client.java:250)
	at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
Any help would be really appreciated!

Thank you!
Free forum by Nabble Edit this page