Bug in Hybrid Hash Join

[Flavio Pompermaier]

Hi guys,
my job fails on Flink 0.10-snapshot with the following message: Bug in Hybrid Hash Join: Request to spill a partition with less than two buffers.I tried to double network buffers but I still got the error :(

Is it a real bug or am I doing something wrong?

See you very soon :)
Flavio

[Sachin Goel]

Hi Flavio
This was addressed in Flink-2763. Can you check if you're using the latest version?

Cheers!
Sachin

On Oct 13, 2015 5:04 AM, "Flavio Pompermaier" <[hidden email]> wrote:

Hi guys,
my job fails on Flink 0.10-snapshot with the following message: Bug in Hybrid Hash Join: Request to spill a partition with less than two buffers.I tried to double network buffers but I still got the error :(

Is it a real bug or am I doing something wrong?

See you very soon :)
Flavio

[Flavio Pompermaier]

I ensured that I was using the latest version and I showed to program to Fabian right now.
The stack trace is:

Caused by: java.lang.RuntimeException: Bug in Hybrid Hash Join: Request to spill a partition with less than two buffers.
at org.apache.flink.runtime.operators.hash.HashPartition.spillPartition(HashPartition.java:301)
at org.apache.flink.runtime.operators.hash.MutableHashTable.spillPartition(MutableHashTable.java:1108)
at org.apache.flink.runtime.operators.hash.MutableHashTable.nextSegment(MutableHashTable.java:1277)
at org.apache.flink.runtime.operators.hash.HashPartition$BuildSideBuffer.nextSegment(HashPartition.java:523)
at org.apache.flink.runtime.memory.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
at org.apache.flink.runtime.memory.AbstractPagedOutputView.write(AbstractPagedOutputView.java:201)
at org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream.write(DataOutputViewStream.java:39)
at com.esotericsoftware.kryo.io.Output.flush(Output.java:163)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:187)
at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:116)
at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:30)
at org.apache.flink.runtime.operators.hash.HashPartition.insertIntoBuildBuffer(HashPartition.java:256)

at org.apache.flink.runtime.operators.hash.MutableHashTable.insertIntoTable(MutableHashTable.java:856)
at org.apache.flink.runtime.operators.hash.MutableHashTable.buildInitialTable(MutableHashTable.java:685)
at org.apache.flink.runtime.operators.hash.MutableHashTable.open(MutableHashTable.java:443)
at org.apache.flink.runtime.operators.hash.NonReusingBuildSecondHashMatchIterator.open(NonReusingBuildSecondHashMatchIterator.java:85)
at org.apache.flink.runtime.operators.JoinDriver.prepare(JoinDriver.java:195)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:459)
... 3 more

On Tue, Oct 13, 2015 at 5:20 AM, Sachin Goel <[hidden email]> wrote:

Hi Flavio
This was addressed in Flink-2763. Can you check if you're using the latest version?

Cheers!
Sachin

On Oct 13, 2015 5:04 AM, "Flavio Pompermaier" <[hidden email]> wrote:

Hi guys,
my job fails on Flink 0.10-snapshot with the following message: Bug in Hybrid Hash Join: Request to spill a partition with less than two buffers.I tried to double network buffers but I still got the error :(

Is it a real bug or am I doing something wrong?

See you very soon :)
Flavio

[Sachin Goel]

Okay. Great!

Please re-open the jira in case the issue hasn't been resolved.

Cheers!

Sachin

On Tue, Oct 13, 2015 at 4:00 PM, Flavio Pompermaier <[hidden email]> wrote:

I ensured that I was using the latest version and I showed to program to Fabian right now.
The stack trace is:

Caused by: java.lang.RuntimeException: Bug in Hybrid Hash Join: Request to spill a partition with less than two buffers.
at org.apache.flink.runtime.operators.hash.HashPartition.spillPartition(HashPartition.java:301)
at org.apache.flink.runtime.operators.hash.MutableHashTable.spillPartition(MutableHashTable.java:1108)
at org.apache.flink.runtime.operators.hash.MutableHashTable.nextSegment(MutableHashTable.java:1277)
at org.apache.flink.runtime.operators.hash.HashPartition$BuildSideBuffer.nextSegment(HashPartition.java:523)
at org.apache.flink.runtime.memory.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
at org.apache.flink.runtime.memory.AbstractPagedOutputView.write(AbstractPagedOutputView.java:201)
at org.apache.flink.api.java.typeutils.runtime.DataOutputViewStream.write(DataOutputViewStream.java:39)
at com.esotericsoftware.kryo.io.Output.flush(Output.java:163)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:187)
at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:116)
at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:30)
at org.apache.flink.runtime.operators.hash.HashPartition.insertIntoBuildBuffer(HashPartition.java:256)

at org.apache.flink.runtime.operators.hash.MutableHashTable.insertIntoTable(MutableHashTable.java:856)
at org.apache.flink.runtime.operators.hash.MutableHashTable.buildInitialTable(MutableHashTable.java:685)
at org.apache.flink.runtime.operators.hash.MutableHashTable.open(MutableHashTable.java:443)
at org.apache.flink.runtime.operators.hash.NonReusingBuildSecondHashMatchIterator.open(NonReusingBuildSecondHashMatchIterator.java:85)
at org.apache.flink.runtime.operators.JoinDriver.prepare(JoinDriver.java:195)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:459)
... 3 more

On Tue, Oct 13, 2015 at 5:20 AM, Sachin Goel <[hidden email]> wrote:

Hi Flavio
This was addressed in Flink-2763. Can you check if you're using the latest version?

Cheers!
Sachin

On Oct 13, 2015 5:04 AM, "Flavio Pompermaier" <[hidden email]> wrote:

Hi guys,
my job fails on Flink 0.10-snapshot with the following message: Bug in Hybrid Hash Join: Request to spill a partition with less than two buffers.I tried to double network buffers but I still got the error :(

Is it a real bug or am I doing something wrong?

See you very soon :)
Flavio