An error occurred in the channel

[Rockstar Flo]

Hi,

I get a runtime error while executing my flink0.6 job.
"An error occurred in the channel"
Could anyone help me?

Greetings
Florian

[Ufuk Celebi]

Hey Florian,

this is a known issue and the commit, which introduced the problem has been reverted for the 0.6.1 release. Could you please work with that version?

– Ufuk

On 08 Oct 2014, at 14:34, Florian Hönicke <[hidden email]> wrote:

> Hi,
>
> I get a runtime error while executing my flink0.6 job.
> "An error occurred in the channel"
> Could anyone help me?
>
> Greetings
> Florian
> <out.txt>

[Rockstar Flo]

Thanks allot.
I switched to flink0.6.1
Now i get a slightly different exception:  "The channel is erroneous."

...
...rojectFlatJoinFunction) (159/162) switched to CANCELED
10/08/2014 20:27:58:
Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction)
(160/162) switched to CANCELED
10/08/2014 20:27:58:
Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction)
(161/162) switched to CANCELED
10/08/2014 20:27:58:    Job execution switched to status FAILED
Error: The program execution failed: java.lang.Exception: The data
preparation for task
'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)' ,
caused an error: Error obtaining the sorted input: Thread 'SortMerger
spilling thread' terminated due to an exception: The channel is erroneous.
         at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
         at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
         at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
         at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
Thread 'SortMerger spilling thread' terminated due to an exception: The
channel is erroneous.
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
         at
org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
         at
org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
         at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
         ... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: The channel is erroneous.
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
Caused by: java.io.IOException: The channel is erroneous.
         at
org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
         at
org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
         at
org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
         at
org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
         at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
         at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
         at
org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
         at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
         at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
Caused by: java.io.IOException: No space left on device
         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
         at
org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
         at
org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)

org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: java.lang.Exception: The data preparation for task
'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)' ,
caused an error: Error obtaining the sorted input: Thread 'SortMerger
spilling thread' terminated due to an exception: The channel is erroneous.
         at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
         at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
         at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
         at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
Thread 'SortMerger spilling thread' terminated due to an exception: The
channel is erroneous.
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
         at
org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
         at
org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
         at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
         ... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: The channel is erroneous.
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
Caused by: java.io.IOException: The channel is erroneous.
         at
org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
         at
org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
         at
org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
         at
org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
         at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
         at
org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
         at
org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
         at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
         at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
         at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
Caused by: java.io.IOException: No space left on device
         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
         at
org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
         at
org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)

         at org.apache.flink.client.program.Client.run(Client.java:325)
         at org.apache.flink.client.program.Client.run(Client.java:291)
         at org.apache.flink.client.program.Client.run(Client.java:285)
         at
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:54)
         at
org.apache.flink.allIn.StratosphereMultiFlink6Job.main(StratosphereMultiFlink6Job.java:72)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:389)
         at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:307)
         at org.apache.flink.client.program.Client.run(Client.java:244)
         at
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:332)
         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
         at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:930)
         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954)


Am 08.10.2014 15:04, schrieb Ufuk Celebi:

> Hey Florian,
>
> this is a known issue and the commit, which introduced the problem has been reverted for the 0.6.1 release. Could you please work with that version?
>
> – Ufuk
>
> On 08 Oct 2014, at 14:34, Florian Hönicke <[hidden email]> wrote:
>
>> Hi,
>>
>> I get a runtime error while executing my flink0.6 job.
>> "An error occurred in the channel"
>> Could anyone help me?
>>
>> Greetings
>> Florian
>> <out.txt>

[Ufuk Celebi]

Hey Florian,

sorry that you run into these annoying problems, but in this case I think it is pretty much what it says and the system is running out of space.

The asynchronous I/O writer thread tries to write to a file channel, which throws the corresponding IOException "No space left on device".

I would suggest that you manually clean up the tmp directories of your machine(s). If you didn't configure anything, Flink falls back to "java.io.tmpdir", which is the system dependent temp file path, e.g. /tmp.

Please keep us updated if this helps or not.

(Flink should actually clean up everything when the disk based operators close. I'm not sure what happens when a program dies with an exception.)

PS: We should check whether we can improve the error message or at least log the used temp directories when the IOManager starts. I will open a corresponding issue.

On 08 Oct 2014, at 20:49, Florian Hönicke <[hidden email]> wrote:

> Thanks allot.
> I switched to flink0.6.1
> Now i get a slightly different exception:  "The channel is erroneous."
>
> ...
> ...rojectFlatJoinFunction) (159/162) switched to CANCELED
> 10/08/2014 20:27:58: Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction) (160/162) switched to CANCELED
> 10/08/2014 20:27:58: Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction) (161/162) switched to CANCELED
> 10/08/2014 20:27:58:    Job execution switched to status FAILED
> Error: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
>        at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
>        at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>        at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
>        at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
>        at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
>        ... 3 more
> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
> Caused by: java.io.IOException: The channel is erroneous.
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
>        at org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
>        at org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
>        at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
>        at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
> Caused by: java.io.IOException: No space left on device
>        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>        at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>        at org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
>        at org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)
>
> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
>        at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
>        at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>        at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
>        at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
>        at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
>        ... 3 more
> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
> Caused by: java.io.IOException: The channel is erroneous.
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
>        at org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
>        at org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
>        at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
>        at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
> Caused by: java.io.IOException: No space left on device
>        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>        at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>        at org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
>        at org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)
>
>        at org.apache.flink.client.program.Client.run(Client.java:325)
>        at org.apache.flink.client.program.Client.run(Client.java:291)
>        at org.apache.flink.client.program.Client.run(Client.java:285)
>        at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:54)
>        at org.apache.flink.allIn.StratosphereMultiFlink6Job.main(StratosphereMultiFlink6Job.java:72)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:389)
>        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:307)
>        at org.apache.flink.client.program.Client.run(Client.java:244)
>        at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:332)
>        at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
>        at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:930)
>        at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954)
>
>
> Am 08.10.2014 15:04, schrieb Ufuk Celebi:
>> Hey Florian,
>>
>> this is a known issue and the commit, which introduced the problem has been reverted for the 0.6.1 release. Could you please work with that version?
>>
>> – Ufuk
>>
>> On 08 Oct 2014, at 14:34, Florian Hönicke <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> I get a runtime error while executing my flink0.6 job.
>>> "An error occurred in the channel"
>>> Could anyone help me?
>>>
>>> Greetings
>>> Florian
>>> <out.txt>
>
> <StratosphereMultiFlink6Job.java><flink-conf.yaml>

[Rockstar Flo]

Problem solved :)
Thanks allot!
I deleted ~200GB tmp-files.
Flink did not clean up everything.

Greetings Florian

Am 08.10.2014 21:14, schrieb Ufuk Celebi:

> Hey Florian,
>
> sorry that you run into these annoying problems, but in this case I think it is pretty much what it says and the system is running out of space.
>
> The asynchronous I/O writer thread tries to write to a file channel, which throws the corresponding IOException "No space left on device".
>
> I would suggest that you manually clean up the tmp directories of your machine(s). If you didn't configure anything, Flink falls back to "java.io.tmpdir", which is the system dependent temp file path, e.g. /tmp.
>
> Please keep us updated if this helps or not.
>
> (Flink should actually clean up everything when the disk based operators close. I'm not sure what happens when a program dies with an exception.)
>
> PS: We should check whether we can improve the error message or at least log the used temp directories when the IOManager starts. I will open a corresponding issue.
>
> On 08 Oct 2014, at 20:49, Florian Hönicke <[hidden email]> wrote:
>
>> Thanks allot.
>> I switched to flink0.6.1
>> Now i get a slightly different exception:  "The channel is erroneous."
>>
>> ...
>> ...rojectFlatJoinFunction) (159/162) switched to CANCELED
>> 10/08/2014 20:27:58: Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction) (160/162) switched to CANCELED
>> 10/08/2014 20:27:58: Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction) (161/162) switched to CANCELED
>> 10/08/2014 20:27:58:    Job execution switched to status FAILED
>> Error: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>>         at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
>>         at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
>>         at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>>         at java.lang.Thread.run(Thread.java:744)
>> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
>>         at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
>>         at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
>>         at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
>>         ... 3 more
>> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
>> Caused by: java.io.IOException: The channel is erroneous.
>>         at org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
>>         at org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
>>         at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
>>         at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
>>         at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
>>         at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
>>         at org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
>>         at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
>>         at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
>> Caused by: java.io.IOException: No space left on device
>>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>>         at org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
>>         at org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)
>>
>> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>>         at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
>>         at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
>>         at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>>         at java.lang.Thread.run(Thread.java:744)
>> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
>>         at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
>>         at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
>>         at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
>>         ... 3 more
>> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The channel is erroneous.
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
>> Caused by: java.io.IOException: The channel is erroneous.
>>         at org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
>>         at org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
>>         at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
>>         at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
>>         at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
>>         at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
>>         at org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
>>         at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
>>         at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
>>         at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
>> Caused by: java.io.IOException: No space left on device
>>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>>         at org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
>>         at org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)
>>
>>         at org.apache.flink.client.program.Client.run(Client.java:325)
>>         at org.apache.flink.client.program.Client.run(Client.java:291)
>>         at org.apache.flink.client.program.Client.run(Client.java:285)
>>         at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:54)
>>         at org.apache.flink.allIn.StratosphereMultiFlink6Job.main(StratosphereMultiFlink6Job.java:72)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:389)
>>         at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:307)
>>         at org.apache.flink.client.program.Client.run(Client.java:244)
>>         at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:332)
>>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
>>         at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:930)
>>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954)
>>
>>
>> Am 08.10.2014 15:04, schrieb Ufuk Celebi:
>>> Hey Florian,
>>>
>>> this is a known issue and the commit, which introduced the problem has been reverted for the 0.6.1 release. Could you please work with that version?
>>>
>>> – Ufuk
>>>
>>> On 08 Oct 2014, at 14:34, Florian Hönicke <[hidden email]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I get a runtime error while executing my flink0.6 job.
>>>> "An error occurred in the channel"
>>>> Could anyone help me?
>>>>
>>>> Greetings
>>>> Florian
>>>> <out.txt>
>> <StratosphereMultiFlink6Job.java><flink-conf.yaml>