We have a data which is broad and slow; hundreds of thousands of
keys, a small number will get an event every few seconds, some get
an event every few days, and the vast majority will get an event
in a few times an hour. Let's say then that keeping this data in
heap for the last couple days is not a challenge. However, one
additional challenge is that we can receive late events or
corrective data, going back indefinitely, and while infrequent, we
need to be able to handle this gracefully. Lets say that the
total data-set grows too large to keep in memory economically.
One approach of course is a "lambda" type, where sufficiently
late events are noted to a side channel, perhaps triggering some
batch job to be scheduled.
However I'm pondering a simpler solution, I understand that
with the RocksDB, the state size can exceed the heap. Would it be
a plausible approach in this situation to never purge windows,
keeping computation state back to "the beginning", so that an
arbitrarily old window (years, potentially), could re-emit a
corrected value?
Thanks
_Derek
Locked
