How to generate a sequential watermark which increases by one unit each time

[Averell]

Hi everyone,

I have a stream of files, each file has multiple records. A record has one
Long field named `ts`, which holds the file creation time (so it increases
over time, and one same value for all records in each file). However, I need
to have a watermark that increases by exactly one unit for each file.

I thought of extending the AssignerWithPeriodicWatermarks interface with a
member variable holding that sequence value. However, it seems to me that it
is not possible to persist that value during checkpoints.

Are there any options for me?

Thanks and best regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

[Paul Lam]

Hi Averell,

IMHO, a simple approach would be adding a rich map that holds the sequence value (backed by states) and attach it to the records before the assigner operator.

Best,

Paul Lam

在 2019年5月21日，20:37，Averell <[hidden email]> 写道：

Hi everyone,

I have a stream of files, each file has multiple records. A record has one
Long field named `ts`, which holds the file creation time (so it increases
over time, and one same value for all records in each file). However, I need
to have a watermark that increases by exactly one unit for each file.

I thought of extending the AssignerWithPeriodicWatermarks interface with a
member variable holding that sequence value. However, it seems to me that it
is not possible to persist that value during checkpoints.

Are there any options for me?

Thanks and best regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/