Encountered unregistered class ID

[Flavio Pompermaier]

Hi to all,

I'm running a job (with Flink 0.10-SNAPSHOT) that reads some parquet-thrift objects and then it performs some joins  and I receive the following exception:

Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Encountered unregistered class ID: 93

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784)

Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 93

at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)

at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)

at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235)

at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71)

at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)

What could be the cause of this?

Best,

Flavio

[Stephan Ewen]

Hi Flavio!

That exception means that the Kryo serializers are not in sync. The writers have registered types that the readers do not know.

Two possible reasons that I can think of from the top of my head:

1) Do you manually register types? Are you registering new types in the middle of your program?

2) Flink may not be passing the Kryo registration properly down to the sort reader.

Greetings,
Stephan

On Wed, Aug 26, 2015 at 9:04 AM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm running a job (with Flink 0.10-SNAPSHOT) that reads some parquet-thrift objects and then it performs some joins  and I receive the following exception:

Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Encountered unregistered class ID: 93

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784)

Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 93

at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)

at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)

at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235)

at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71)

at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)

What could be the cause of this?

Best,

Flavio

[Flavio Pompermaier]

I'm running my job from my Eclipse and I don't register any Kryo class in the env.

On Wed, Aug 26, 2015 at 10:34 AM, Stephan Ewen <[hidden email]> wrote:

Hi Flavio!

That exception means that the Kryo serializers are not in sync. The writers have registered types that the readers do not know.

Two possible reasons that I can think of from the top of my head:

1) Do you manually register types? Are you registering new types in the middle of your program?

2) Flink may not be passing the Kryo registration properly down to the sort reader.

Greetings,
Stephan

On Wed, Aug 26, 2015 at 9:04 AM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm running a job (with Flink 0.10-SNAPSHOT) that reads some parquet-thrift objects and then it performs some joins  and I receive the following exception:

Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Encountered unregistered class ID: 93

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784)

Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 93

at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)

at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)

at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235)

at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71)

at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)

What could be the cause of this?

Best,

Flavio

[rmetzger0]

Hi Flavio,

can you share a minimal version of your program to reproduce the issue?

On Wed, Aug 26, 2015 at 10:36 AM, Flavio Pompermaier <[hidden email]> wrote:

I'm running my job from my Eclipse and I don't register any Kryo class in the env.

On Wed, Aug 26, 2015 at 10:34 AM, Stephan Ewen <[hidden email]> wrote:

Hi Flavio!

That exception means that the Kryo serializers are not in sync. The writers have registered types that the readers do not know.

Two possible reasons that I can think of from the top of my head:

1) Do you manually register types? Are you registering new types in the middle of your program?

2) Flink may not be passing the Kryo registration properly down to the sort reader.

Greetings,
Stephan

On Wed, Aug 26, 2015 at 9:04 AM, Flavio Pompermaier <[hidden email]> wrote:

Hi to all,

I'm running a job (with Flink 0.10-SNAPSHOT) that reads some parquet-thrift objects and then it performs some joins  and I receive the following exception:

Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Encountered unregistered class ID: 93

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784)

Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 93

at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)

at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)

at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225)

at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235)

at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71)

at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328)

at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)

What could be the cause of this?

Best,

Flavio