How to load Avro file in a Dataset

[Soheil Pourbafrani]

According to the Flink document, it's possible to load Avro file like the following:

AvroInputFormat<User> users = new AvroInputFormat<User>(in, User.class);
DataSet<User> usersDS = env.createInput(users);

It's a bit confusing for me. I guess the User is a predefined class. My question is can Flink detect the Avro file schema automatically? How can I load Avro file without any predefined class?

[Ken Krugler]

Hi Soheil,

I’ve used Avro in the past, but I’m no expert - so I could be missing something obvious here…

But if you don’t know any of the fields in the schema, then what processing would you do with the data in your Flink workflow?

— Ken

On Jan 27, 2019, at 5:50 AM, Soheil Pourbafrani <[hidden email]> wrote:

According to the Flink document, it's possible to load Avro file like the following:

AvroInputFormat<User> users = new AvroInputFormat<User>(in, User.class);
DataSet<User> usersDS = env.createInput(users);

It's a bit confusing for me. I guess the User is a predefined class. My question is can Flink detect the Avro file schema automatically? How can I load Avro file without any predefined class?

--------------------------

Ken Krugler
[phone]
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra