Thank you, that is very helpful.
This is a thought experiment do bear with me.
Our objective is to find paths ( funnel analysis) in a sessionized stream. The source streams ( kafka partitions sources) have been watermarked. A path a A[*]B as in start at A and ends at B, a funnel to be more precise.
If your contention is true, and that is, that the water mark in the TriggerContext
is indeed a reflection of event time based progress of the stream, than emitting using a Time Based Tigger say every n seconds of WM progression, indeed implies that the window will have received all events before up till the operators sense of the watermark from various incident sources and this approximate order, that though elements within the batch between emits may be unordered, the batches themselves are ordered, can be exploited. We would essentially do a FIRE ( no purge ) every n seconds, look at the items within the the pane, reduce the elements by extracting paths
realized, trimming the elements fully or up till a dangling A ( B has not appeared in the stream ), drastically reducing the state size. ( A[*}B are not numerous, they are very specific funnels registration etc )
That begs a question ( though I would be disappointed if that is not the case )
Does Time Based Trigger fire based on it's sense of Watermark as in say we wish to fire every 1 minute, the firing is the WMs progress of a minute rather than wall clock minute. If that is not the case we will have to write a custom trigger?
Thank you for helping out.
Locked
