Re: Serialization questions

Posted by Flavio Pompermaier on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Serialization-questions-tp21561p21592.html

Hi Minglei,

using the registerTypeWithKryoSerializer with the 3 classes works (without disableGenericTypes) but the problem is that I would like to avoid Kryo serialization if this is useful to speedup the job performance,

and thus I'd like to be able to run all jobs with disableGenericTypes. 

Best,

Flavio

On Wed, Jul 18, 2018 at 11:10 AM, zhangminglei <[hidden email]> wrote:

Hi, Flavio

• addDefaultKryoSerializer differs from registerTypeWithKryoSerializer because addDefaultKryoSerializer use the passed serializer also for subclasses of the configured class. Am I right? This is not very clear in the method's Javadoc…

I think it is not exactly a problem with flink. Instead of a kryo problem. For example, addDefaultKryoSerializer corresponding to the addDefaultSerializer(int[].class, IntArraySerializer.class) in kryo, whereas registerTypeWithKryoSerializer corresponding to the register(int.class, new IntSerializer()) in kryo.With register, you explicitly assign an id for that type plus serializer. The default serializer just tells kryo which serializer to use when this type has to be serialized, kryo will then implicitly register the serializer. And the advantage of using register would be [1]. when setting setRegistrationRequired(true), which is recommended (and will be the default in 5.0), you'd have to register every occurring type explicitly.

• how to avoid that exception?

You can try below and do not make disableGenericTypes and see what happens.

env.registerTypeWithKryoSerializer(DateTime.class, JodaDateTimeSerializer.class);

env.registerTypeWithKryoSerializer(EntitonAtom.class, TBaseSerializer.class);

env.registerTypeWithKryoSerializer(EntitonQuad.class, TBaseSerializer.class);

[1] https://github.com/EsotericSoftware/kryo/blob/master/README.md#registration 

Cheers

Minglei

在 2018年7月17日，下午9:00，Flavio Pompermaier <[hidden email]> 写道：

Hi to all,

I was trying to check whether our jobs are properly typed or not.

I've started disabling generic types[1] in order to discover untyped transformations and so I added the proper returns() to operators.

Unfortunately there are jobs where we serialize Thrift and DateTime objects, so I need to properly configure the serializers in the ExecutionEnvironment:

env.registerTypeWithKryoSerializer(DateTime.class, JodaDateTimeSerializer.class);

env.getConfig().addDefaultKryoSerializer(EntitonAtom.class, TBaseSerializer.class);

env.getConfig().addDefaultKryoSerializer(EntitonQuad.class, TBaseSerializer.class);

Those jobs don't work when I disable generic types and I get the following exception:

Exception in thread "main" java.lang.UnsupportedOperationException: Generic types have been 

disabled in the ExecutionConfig and type xxx.EntitonAtom is treated as a generic type.

 I have a couple of questions:

• addDefaultKryoSerializer differs from registerTypeWithKryoSerializer because addDefaultKryoSerializer use the passed serializer also for subclasses of the configured class. Am I right? This is not very clear in the method's Javadoc...
• how to avoid that exception?

Best,

Flavio

[1] env.getConfig().disableGenericTypes();