TypeSerializer Example

[Sandeep khanzode]

Hello,

Is there a working example of a TypeSerializer for a Java type stored in the State?

My requirement is that I should be able to store the Java POJO entity in the MapState. The state is backed by RocksDBBackend.

If I update the entity with a new member variable, I am unable to deserialise the state into the new entity. 

I checked this link.

https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html

It does mention that the POJO type is special based on the rules. Does that mean that I can add or remove member variables for the POJO? I have been unable to get it to work. 

Thanks,

Sandeep

[Timo Walther]

Hi Sandeep,

did you have a chance to look at this documentation page?

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/custom_serialization.html

The interfaces might not be easy to implement but are very powerful to
address compatibility issues. You can also look into Flink serializers
for some examples:

https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime

Esp:

https://github.com/apache/flink/blob/89c6c03660a88a648bbd13b4e6696124fe46d013/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java#L599

For the POJO logic.

By the way, usually we recommend Avro for state data structures if
schema evolution is a topic.

Regards,
Timo



On 29.04.21 18:10, Sandeep khanzode wrote:

> Hello,
>
> Is there a working example of a TypeSerializer for a Java type stored in
> the State?
>
> My requirement is that I should be able to store the Java POJO entity in
> the MapState. The state is backed by RocksDBBackend.
>
> If I update the entity with a new member variable, I am unable to
> deserialise the state into the new entity.
>
> I checked this link.
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html 
> <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html>
>
> It does mention that the POJO type is special based on the rules. Does
> that mean that I can add or remove member variables for the POJO? I have
> been unable to get it to work.
>
>
> Thanks,
> Sandeep

[Sandeep khanzode]

Hi Timo,

Thanks! I will take a look at the links.

Can you please share if you have any simple (or complex) example of Avro state data structures?

Thanks,
Sandeep

> On 30-Apr-2021, at 4:46 PM, Timo Walther <[hidden email]> wrote:
>
> Hi Sandeep,
>
> did you have a chance to look at this documentation page?
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/custom_serialization.html
>
> The interfaces might not be easy to implement but are very powerful to address compatibility issues. You can also look into Flink serializers for some examples:
>
> https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime
>
> Esp:
>
> https://github.com/apache/flink/blob/89c6c03660a88a648bbd13b4e6696124fe46d013/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java#L599
>
> For the POJO logic.
>
> By the way, usually we recommend Avro for state data structures if schema evolution is a topic.
>
> Regards,
> Timo
>
>
>
> On 29.04.21 18:10, Sandeep khanzode wrote:
>> Hello,
>> Is there a working example of a TypeSerializer for a Java type stored in the State?
>> My requirement is that I should be able to store the Java POJO entity in the MapState. The state is backed by RocksDBBackend.
>> If I update the entity with a new member variable, I am unable to deserialise the state into the new entity.
>> I checked this link.
>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html>
>> It does mention that the POJO type is special based on the rules. Does that mean that I can add or remove member variables for the POJO? I have been unable to get it to work.
>> Thanks,
>> Sandeep
>

[Timo Walther]

I also found these pages:

https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html

https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html#avro

I hope this helps.

Regards,
Timo


On 30.04.21 13:20, Sandeep khanzode wrote:

> Hi Timo,
>
> Thanks! I will take a look at the links.
>
> Can you please share if you have any simple (or complex) example of Avro state data structures?
>
> Thanks,
> Sandeep
>
>> On 30-Apr-2021, at 4:46 PM, Timo Walther <[hidden email]> wrote:
>>
>> Hi Sandeep,
>>
>> did you have a chance to look at this documentation page?
>>
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/custom_serialization.html
>>
>> The interfaces might not be easy to implement but are very powerful to address compatibility issues. You can also look into Flink serializers for some examples:
>>
>> https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime
>>
>> Esp:
>>
>> https://github.com/apache/flink/blob/89c6c03660a88a648bbd13b4e6696124fe46d013/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java#L599
>>
>> For the POJO logic.
>>
>> By the way, usually we recommend Avro for state data structures if schema evolution is a topic.
>>
>> Regards,
>> Timo
>>
>>
>>
>> On 29.04.21 18:10, Sandeep khanzode wrote:
>>> Hello,
>>> Is there a working example of a TypeSerializer for a Java type stored in the State?
>>> My requirement is that I should be able to store the Java POJO entity in the MapState. The state is backed by RocksDBBackend.
>>> If I update the entity with a new member variable, I am unable to deserialise the state into the new entity.
>>> I checked this link.
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html>
>>> It does mention that the POJO type is special based on the rules. Does that mean that I can add or remove member variables for the POJO? I have been unable to get it to work.
>>> Thanks,
>>> Sandeep
>>
>

[Sandeep khanzode]

Hi,

Is there a working example somewhere that I can refer for writing Avro entities in Flink state as well as Avro serializaition in KafkaConsumer/Producer?

I tried to use Avro entities directly but there is an issue beyond Apache Avro 1.7.7 in that the entities created have a serialVersionUid. So when I tried to test schema evolution by adding a member, there was the java serialization issue saying the two generated classes’ serialVersionUids do not match i.e. the one stored in the state and the one being used with the new member variable now.

Is there any configuration that overrides this?

Request you to please provide some references of samples. Thanks.

Thanks,
Sandeep

> On 30-Apr-2021, at 5:00 PM, Timo Walther <[hidden email]> wrote:
>
> I also found these pages:
>
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html
>
> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html#avro
>
> I hope this helps.
>
> Regards,
> Timo
>
>
> On 30.04.21 13:20, Sandeep khanzode wrote:
>> Hi Timo,
>> Thanks! I will take a look at the links.
>> Can you please share if you have any simple (or complex) example of Avro state data structures?
>> Thanks,
>> Sandeep
>>> On 30-Apr-2021, at 4:46 PM, Timo Walther <[hidden email]> wrote:
>>>
>>> Hi Sandeep,
>>>
>>> did you have a chance to look at this documentation page?
>>>
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/custom_serialization.html
>>>
>>> The interfaces might not be easy to implement but are very powerful to address compatibility issues. You can also look into Flink serializers for some examples:
>>>
>>> https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime
>>>
>>> Esp:
>>>
>>> https://github.com/apache/flink/blob/89c6c03660a88a648bbd13b4e6696124fe46d013/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java#L599
>>>
>>> For the POJO logic.
>>>
>>> By the way, usually we recommend Avro for state data structures if schema evolution is a topic.
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>>
>>> On 29.04.21 18:10, Sandeep khanzode wrote:
>>>> Hello,
>>>> Is there a working example of a TypeSerializer for a Java type stored in the State?
>>>> My requirement is that I should be able to store the Java POJO entity in the MapState. The state is backed by RocksDBBackend.
>>>> If I update the entity with a new member variable, I am unable to deserialise the state into the new entity.
>>>> I checked this link.
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html>
>>>> It does mention that the POJO type is special based on the rules. Does that mean that I can add or remove member variables for the POJO? I have been unable to get it to work.
>>>> Thanks,
>>>> Sandeep
>>>
>

[Sandeep khanzode]

Hello,

Can someone please assist for this query? Thanks!

Thanks,
Sandeep

> On 06-May-2021, at 10:30 AM, Sandeep khanzode <[hidden email]> wrote:
>
> Hi,
>
> Is there a working example somewhere that I can refer for writing Avro entities in Flink state as well as Avro serializaition in KafkaConsumer/Producer?
>
> I tried to use Avro entities directly but there is an issue beyond Apache Avro 1.7.7 in that the entities created have a serialVersionUid. So when I tried to test schema evolution by adding a member, there was the java serialization issue saying the two generated classes’ serialVersionUids do not match i.e. the one stored in the state and the one being used with the new member variable now.
>
> Is there any configuration that overrides this?
>
> Request you to please provide some references of samples. Thanks.
>
> Thanks,
> Sandeep
>
>
>> On 30-Apr-2021, at 5:00 PM, Timo Walther <[hidden email]> wrote:
>>
>> I also found these pages:
>>
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/schema_evolution.html
>>
>> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html#avro
>>
>> I hope this helps.
>>
>> Regards,
>> Timo
>>
>>
>> On 30.04.21 13:20, Sandeep khanzode wrote:
>>> Hi Timo,
>>> Thanks! I will take a look at the links.
>>> Can you please share if you have any simple (or complex) example of Avro state data structures?
>>> Thanks,
>>> Sandeep
>>>> On 30-Apr-2021, at 4:46 PM, Timo Walther <[hidden email]> wrote:
>>>>
>>>> Hi Sandeep,
>>>>
>>>> did you have a chance to look at this documentation page?
>>>>
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/custom_serialization.html
>>>>
>>>> The interfaces might not be easy to implement but are very powerful to address compatibility issues. You can also look into Flink serializers for some examples:
>>>>
>>>> https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime
>>>>
>>>> Esp:
>>>>
>>>> https://github.com/apache/flink/blob/89c6c03660a88a648bbd13b4e6696124fe46d013/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java#L599
>>>>
>>>> For the POJO logic.
>>>>
>>>> By the way, usually we recommend Avro for state data structures if schema evolution is a topic.
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>>
>>>> On 29.04.21 18:10, Sandeep khanzode wrote:
>>>>> Hello,
>>>>> Is there a working example of a TypeSerializer for a Java type stored in the State?
>>>>> My requirement is that I should be able to store the Java POJO entity in the MapState. The state is backed by RocksDBBackend.
>>>>> If I update the entity with a new member variable, I am unable to deserialise the state into the new entity.
>>>>> I checked this link.
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/types_serialization.html>
>>>>> It does mention that the POJO type is special based on the rules. Does that mean that I can add or remove member variables for the POJO? I have been unable to get it to work.
>>>>> Thanks,
>>>>> Sandeep
>>>>
>>
>