FlinkKafkaConsumer - Save watermark assigner in the state

[Juan Gentile]

Hello,

 

We are currently facing an issue where we need to store the instance of the watermark and timestamp assigner in the state while consumer from Kafka.

For that purpose we took a look at FlinkKafkaConsumerBase and noticed that since the methods (snapshotState and initializeState from the CheckpointedFunction) are final we can’t override them.

An alternative could be to wrap the class but we don’t like that solution either as it would make it harder to migrate in the future if there are changes there.

We would like to know if it’d be possible to store the ‘assigner’ in the state of the FlinkKafkaConsumerBase. We think it’s not a big change and we are willing to do it if you agree.

 

Thank you,

Juan G.

[Konstantin Knauf-2]

Hi Juan,

can you elaborate a bit on why you want to put the WatermarkAssigner itself into state? It is generally unusual to store a UDF in Managed State.

Cheers,

Konstantin

On Fri, Jul 5, 2019 at 5:07 PM Juan Gentile <[hidden email]> wrote:

Hello,

 

We are currently facing an issue where we need to store the instance of the watermark and timestamp assigner in the state while consumer from Kafka.

For that purpose we took a look at FlinkKafkaConsumerBase and noticed that since the methods (snapshotState and initializeState from the CheckpointedFunction) are final we can’t override them.

An alternative could be to wrap the class but we don’t like that solution either as it would make it harder to migrate in the future if there are changes there.

We would like to know if it’d be possible to store the ‘assigner’ in the state of the FlinkKafkaConsumerBase. We think it’s not a big change and we are willing to do it if you agree.

 

Thank you,

Juan G.

--