Why is IdentityObjectIntMap.get called so often?

[Dmitry Golubets]

Hi,

I've just added my custom MsgPack serializers hoping to see performance increase. I covered all data types in between chains.

However this Kryo method still takes a lot of CPU: IdentityObjectIntMap.get

Is there something else should be configured?
Or is there no way to get away from Kryo overhead?

[Stephan Ewen]

Hi!

I think the best way to get away from Kryo is to write types that go through Flink's own serialization stack:

Have a look here for a bit of background: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/types_serialization.html#flinks-typeinformation-class

BTW: Is the "hot map" involved in Kryo's reference resolution? If yes, you could try a custom hack and see whether it performs better when reference resolution is deactivated (modify the KryoSerializer.checkKryoInitialized() method to deactivate reference handling).

Hope that helps,

Stephan

On Tue, Jan 24, 2017 at 12:47 PM, Dmitry Golubets <[hidden email]> wrote:

Hi,

I've just added my custom MsgPack serializers hoping to see performance increase. I covered all data types in between chains.

However this Kryo method still takes a lot of CPU: IdentityObjectIntMap.get

Is there something else should be configured?
Or is there no way to get away from Kryo overhead?