Data type serialization and testing

Posted by Dave Maughan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Data-type-serialization-and-testing-tp43375.html

Hi,

I recently encountered a scenario where the data type being passed between operators in my streaming job was modified such that it broke serialization. This was due to a non-Avro top-level data type containing an Avro field. The existing integration test (mini cluster) continued to work and unit tests that attempted to cover Kryo serialization continued to work, but when deployed to a real cluster it failed. The problem was easily solved but in future I'd like to catch problems like this in my testing.

Is there a way to force serialization always between all operators in the mini-cluster? Or is there another strategy I can apply to exercise the serialization of my data types?

Thanks,

Dave