(DEPRECATED) Apache Flink User Mailing List archive.

Stack overflow from self referencing Avro schema

classic Classic list List threaded Threaded
♦
♦
 8 messages Options
David Kim
Reply | Threaded
Open this post in threaded view
♦
♦
|

Stack overflow from self referencing Avro schema

David Kim
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David
David Kim
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stack overflow from self referencing Avro schema

David Kim
Hello!

Just wanted to check up on this again. Has anyone else seen this before or have any suggestions?

Thanks!
David

On Tue, Mar 8, 2016 at 12:12 PM, David Kim <[hidden email]> wrote:
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David



--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.
Niels Basjes
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stack overflow from self referencing Avro schema

Niels Basjes
Hi,

Please try to reproduce the problem in a simple (commandline) Java application (i.e. without Flink and such, just Avro).
If you can reproduce it with Avro 1.8.0 then please file a bug report (preferable with the simplest reproduction path you can come up with) via.

Thanks

Niels Basjes

On Thu, Mar 10, 2016 at 4:11 PM, David Kim <[hidden email]> wrote:
Hello!

Just wanted to check up on this again. Has anyone else seen this before or have any suggestions?

Thanks!
David

On Tue, Mar 8, 2016 at 12:12 PM, David Kim <[hidden email]> wrote:
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David



--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.



--
Best regards / Met vriendelijke groeten,

Niels Basjes
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stack overflow from self referencing Avro schema

Stephan Ewen
In reply to this post by David Kim
Hi!

I think that is a TypeExtractor bug. It may actually be a bug for all recursive types.
Let's check this and come up with a fix...

Greetings,
Stephan


On Thu, Mar 10, 2016 at 4:11 PM, David Kim <[hidden email]> wrote:
Hello!

Just wanted to check up on this again. Has anyone else seen this before or have any suggestions?

Thanks!
David

On Tue, Mar 8, 2016 at 12:12 PM, David Kim <[hidden email]> wrote:
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David



--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.
Stephan Ewen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stack overflow from self referencing Avro schema

Stephan Ewen
The following issue should track that. https://issues.apache.org/jira/browse/FLINK-3602

@Niels: Thanks for looking into this. At this point, I think it may actually be a Flink issue, since it concerns the interaction of Avro and Flink's TypeInformation.

On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <[hidden email]> wrote:
Hi!

I think that is a TypeExtractor bug. It may actually be a bug for all recursive types.
Let's check this and come up with a fix...

Greetings,
Stephan


On Thu, Mar 10, 2016 at 4:11 PM, David Kim <[hidden email]> wrote:
Hello!

Just wanted to check up on this again. Has anyone else seen this before or have any suggestions?

Thanks!
David

On Tue, Mar 8, 2016 at 12:12 PM, David Kim <[hidden email]> wrote:
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David



--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.
David Kim
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stack overflow from self referencing Avro schema

David Kim
Thanks Stephan! :)

On Thu, Mar 10, 2016 at 11:06 AM, Stephan Ewen <[hidden email]> wrote:
The following issue should track that. https://issues.apache.org/jira/browse/FLINK-3602

@Niels: Thanks for looking into this. At this point, I think it may actually be a Flink issue, since it concerns the interaction of Avro and Flink's TypeInformation.

On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <[hidden email]> wrote:
Hi!

I think that is a TypeExtractor bug. It may actually be a bug for all recursive types.
Let's check this and come up with a fix...

Greetings,
Stephan


On Thu, Mar 10, 2016 at 4:11 PM, David Kim <[hidden email]> wrote:
Hello!

Just wanted to check up on this again. Has anyone else seen this before or have any suggestions?

Thanks!
David

On Tue, Mar 8, 2016 at 12:12 PM, David Kim <[hidden email]> wrote:
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David



--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.





--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.
rmetzger0
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stack overflow from self referencing Avro schema

rmetzger0
Hey David,

FLINK-3602 has been merged to master.

On Fri, Mar 11, 2016 at 5:11 PM, David Kim <[hidden email]> wrote:
Thanks Stephan! :)

On Thu, Mar 10, 2016 at 11:06 AM, Stephan Ewen <[hidden email]> wrote:
The following issue should track that. https://issues.apache.org/jira/browse/FLINK-3602

@Niels: Thanks for looking into this. At this point, I think it may actually be a Flink issue, since it concerns the interaction of Avro and Flink's TypeInformation.

On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <[hidden email]> wrote:
Hi!

I think that is a TypeExtractor bug. It may actually be a bug for all recursive types.
Let's check this and come up with a fix...

Greetings,
Stephan


On Thu, Mar 10, 2016 at 4:11 PM, David Kim <[hidden email]> wrote:
Hello!

Just wanted to check up on this again. Has anyone else seen this before or have any suggestions?

Thanks!
David

On Tue, Mar 8, 2016 at 12:12 PM, David Kim <[hidden email]> wrote:
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David



--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.





--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.
David Kim
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: Stack overflow from self referencing Avro schema

David Kim
Hi Robert!

Thank you! :)

David

On Tue, Mar 22, 2016 at 7:59 AM, Robert Metzger <[hidden email]> wrote:
Hey David,

FLINK-3602 has been merged to master.

On Fri, Mar 11, 2016 at 5:11 PM, David Kim <[hidden email]> wrote:
Thanks Stephan! :)

On Thu, Mar 10, 2016 at 11:06 AM, Stephan Ewen <[hidden email]> wrote:
The following issue should track that. https://issues.apache.org/jira/browse/FLINK-3602

@Niels: Thanks for looking into this. At this point, I think it may actually be a Flink issue, since it concerns the interaction of Avro and Flink's TypeInformation.

On Thu, Mar 10, 2016 at 6:00 PM, Stephan Ewen <[hidden email]> wrote:
Hi!

I think that is a TypeExtractor bug. It may actually be a bug for all recursive types.
Let's check this and come up with a fix...

Greetings,
Stephan


On Thu, Mar 10, 2016 at 4:11 PM, David Kim <[hidden email]> wrote:
Hello!

Just wanted to check up on this again. Has anyone else seen this before or have any suggestions?

Thanks!
David

On Tue, Mar 8, 2016 at 12:12 PM, David Kim <[hidden email]> wrote:
Hello all,

I'm running into a StackOverflowError using flink 1.0.0. I have an Avro schema that has a self reference. For example:

item.avsc

{
  "namespace": "..."
  "type": "record"
  "name": "Item",
  "fields": [
    {
      "name": "parent"
      "type": ["null, "Item"]
    }
  ]
}

When running my flink job, I'm running into the follow error:

Exception in thread "Thread-94" java.lang.StackOverflowError
	at org.apache.flink.api.java.typeutils.TypeExtractor.countTypeInHierarchy(TypeExtractor.java:1105)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1397)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:48)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1394)
	at org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1319)
	at org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:609)
	at org.apache.flink.api.java.typeutils.TypeExtractor.analyzePojo(TypeExtractor.java:1531)
	at org.apache.flink.api.java.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:53)

Interestingly if I change the type to an Avro array in the schema, this error is not thrown.

Thanks!
David



--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.





--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.




--
Note: this information is confidential. It is prohibited to share, post online or otherwise publicize without Braintree's prior written consent.
Free forum by Nabble Edit this page