Objects with fields that are not serializable

[Dominik Wosiński]

Hey, 

I have a question about using classes with fields that are not serializable in DataStream. Basically, I would like to use the  Java's Optional in DataStream. So Say I have a class Data that has several optional fields and I would like to have DataStream<Data>. I don't think this should cause any issues, but I thought it can be good to ask whether this can cause any issues with Flink Jobs.

Thanks,

Best,

Dom.

[Timo Walther]

Hi Dominik,

Flink does not use Java serialization logic for network communication.
So objects must not implement `Serializable` for usage during runtime
(DataStream<Data>).

Only if those classes are member variables of a Function like
MapFunction, they need to serializable to ship the function code to the
cluster.

You can check which serializer and nested serializers are used by calling:

dataStream.getType().createTypeSerializer(env.getExecutionConfig)

Regards,
Timo



On 14.04.20 12:55, Dominik Wosiński wrote:

> Hey,
> I have a question about using classes with fields that are not
> serializable in DataStream. Basically, I would like to use the  Java's
> Optional in DataStream. So Say I have a class /Data /that has several
> optional fields and I would like to have /DataStream<Data>/. I don't
> think this should cause any issues, but I thought it can be good to ask
> whether this can cause any issues with Flink Jobs.
>
> Thanks,
> Best,
> Dom.