Flink Key based watermarks with event time

[Gnanasoundari Soundarajan]

Hi all,

 

We have a requirement where we need to maintain key based watermarks with event time. Each sensor will communicate with different timestamp where we need to maintain watermark separately for each sensor. Is this requirement can be achieved with Flink?

 

Thanks.

 

Regards,

Gnana

[Congxian Qiu]

Hi

Maybe you can try KeyedProcessFunction[1] for this, but you need to handle the allow-latency logic[2] in your own business logic(event-time records maybe out-of-order)

[1]  https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/process_function.html#the-keyedprocessfunction
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/windows.html#allowed-lateness

Gnanasoundari Soundarajan <[hidden email]> 于2020年5月14日周四 下午12:10写道：

Hi all,

 

We have a requirement where we need to maintain key based watermarks with event time. Each sensor will communicate with different timestamp where we need to maintain watermark separately for each sensor. Is this requirement can be achieved with Flink?

 

Thanks.

 

Regards,

Gnana

[Arvid Heise-3]

Watermarks are maintained for each source separately, but need to be combined when you combine the different streams. If not, you would not be able to perform windows and any related join or aggregation. Having different timestamps in the sources is actually the normal case and the main motivation to use event time altogether.

Btw Watermarks should be viewed as purely technical. You wouldn't explicitly put it in the datastream, but it's maintained by the system while the generation logic is provided by users.

From your description, it feels like you have a different understanding of watermarks. It would help if you line out your use case in more detail.

On Fri, May 15, 2020 at 1:17 PM Congxian Qiu <[hidden email]> wrote:

Hi

Maybe you can try KeyedProcessFunction[1] for this, but you need to handle the allow-latency logic[2] in your own business logic(event-time records maybe out-of-order)

[1]  https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/process_function.html#the-keyedprocessfunction
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/windows.html#allowed-lateness

Best,

Congxian

Gnanasoundari Soundarajan <[hidden email]> 于2020年5月14日周四 下午12:10写道：

Hi all,

 

We have a requirement where we need to maintain key based watermarks with event time. Each sensor will communicate with different timestamp where we need to maintain watermark separately for each sensor. Is this requirement can be achieved with Flink?

 

Thanks.

 

Regards,

Gnana

--