Update avro to 1.7.7 or later for flink 1.1.4

[Torok, David]

I spent close to two days and tracked down the solution to a major issue with Avro / GenericRecord and Flink.  In short, there is a field marked ‘transient’ in Avro 1.7.6 and earlier which interferes with correct Kryo serialization.  This was fixed in Avro 1.7.7, but Flink is dependent on Avro 1.7.6 in its POM.xml file.

 

I recorded the root cause and solution in JIRA https://issues.apache.org/jira/browse/FLINK-5039

 

However the issue is marked as ‘minor’ I’d like a little more attention and hopefully the Avro version can be updated for Flink 1.1.4?

 

In the meantime, I have created a custom Flink distribution Jar containing the Avro 1.7.7 classfiles and it is working perfectly for me now.

 

Best Regards,

Dave Torok

[Timo Walther]

Hi David,

thanks for looking into this. Since you already looked into this issue and solved/tested your fix, it would be great if you could open a pull request for it. Every contribution is very welcome.

Regards,
Timo


Am 04/12/16 um 23:35 schrieb Torok, David:

I spent close to two days and tracked down the solution to a major issue with Avro / GenericRecord and Flink.  In short, there is a field marked ‘transient’ in Avro 1.7.6 and earlier which interferes with correct Kryo serialization.  This was fixed in Avro 1.7.7, but Flink is dependent on Avro 1.7.6 in its POM.xml file.

 

I recorded the root cause and solution in JIRA https://issues.apache.org/jira/browse/FLINK-5039

 

However the issue is marked as ‘minor’ I’d like a little more attention and hopefully the Avro version can be updated for Flink 1.1.4?

 

In the meantime, I have created a custom Flink distribution Jar containing the Avro 1.7.7 classfiles and it is working perfectly for me now.

 

Best Regards,

Dave Torok