AVRO Union type support in Flink

[vishnuviswanath]

Hi All,

Does Flink support AVRO union types - Documentation says it supports nullable types: {"name": "type_double_test", "type": ["null", "double"]}

But my schema has something like : {"name": "union_field", "type": ["string", "double"]}

Thanks

Vishnu

[Martin Eden]

Hey Vishnu,

For those of us on the list that are not very familiar with Flink and Avro can you give a pointed to the docs you are referring to and how you intend to use it? Just so we gain understanding as well.

Thanks,

Martin

On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath <[hidden email]> wrote:

Hi All,

Does Flink support AVRO union types - Documentation says it supports nullable types: {"name": "type_double_test", "type": ["null", "double"]}

But my schema has something like : {"name": "union_field", "type": ["string", "double"]}

Thanks

Vishnu

[Timo Walther]

Hi Vishnu,

I took a look into the code. Actually, we should support it. However, those types might be mapped to Java Objects that will be serialized with our generic Kryo serializer. Have you tested it?

Regards,
Timo


Am 19.07.17 um 06:30 schrieb Martin Eden:

Hey Vishnu,

For those of us on the list that are not very familiar with Flink and Avro can you give a pointed to the docs you are referring to and how you intend to use it? Just so we gain understanding as well.

Thanks,

Martin

On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath <[hidden email]> wrote:

Hi All,

Does Flink support AVRO union types - Documentation says it supports nullable types: {"name": "type_double_test", "type": ["null", "double"]}

But my schema has something like : {"name": "union_field", "type": ["string", "double"]}

Thanks

Vishnu

[vishnuviswanath]

Hi Timo,

I just modified AvroOutputFormatTest to test this and it works fine!. I don't plan to use it to key by, but it is a good point. Thanks.

Regards,

Vishnu

On Wed, Jul 19, 2017 at 10:57 AM, Timo Walther <[hidden email]> wrote:

We have similar checks in our KafkaAvroTableSource, but I could not find such a check in AvroTypeInfo. The field should have a generic type, so you can work with it. If you want to use it as key, you might have to use a mapper before and convert it into a valid key type.

Timo

-------- Weitergeleitete Nachricht --------

Betreff:	Re: AVRO Union type support in Flink
Datum:	Wed, 19 Jul 2017 10:26:24 -0400
Von:	Vishnu Viswanath [hidden email]
An:	Timo Walther [hidden email]

Hi Timo,

Thanks for checking that. I did not try yet. My current application uses Cascading and it has the limitation that Union cannot contain two concrete types - link, so was wondering if I can use Flink. Will give it a try. 

Hi Martin,

The documentation is here

I use it to create AVRO files from source files in S3 and write to Kafka.

Thanks,

Vishnu

On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther <[hidden email]> wrote:

Hi Vishnu,

I took a look into the code. Actually, we should support it. However, those types might be mapped to Java Objects that will be serialized with our generic Kryo serializer. Have you tested it?

Regards,
Timo


Am 19.07.17 um 06:30 schrieb Martin Eden:

Hey Vishnu,

For those of us on the list that are not very familiar with Flink and Avro can you give a pointed to the docs you are referring to and how you intend to use it? Just so we gain understanding as well.

Thanks,

Martin

On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath <[hidden email]> wrote:

Hi All,

Does Flink support AVRO union types - Documentation says it supports nullable types: {"name": "type_double_test", "type": ["null", "double"]}

But my schema has something like : {"name": "union_field", "type": ["string", "double"]}

Thanks

Vishnu