We collect driving data from thousands of users, each vehicle is associated
with a IMEI (unique code). The device installed in these vehicles emits GPS
points in 5 second intervals. My requirement is to assemble all the GPS
points that belong to a single trip and construct a Trip object, for a given
IMEI.
I am using event time and Session windows to detect the end of a trip (10
minutes of non reception of GPS coordinates), and another 15 minutes of
allowed lateness to wait for late events. The watermark then gets advanced
to the latest received event time. Let's say this is for IMEI=100. Now, if I
receive data for IMEI's 1 through 99 that have event time behind this
watermark, all that data will be deemed late and won't be processed.
In other words, if one vehicle's data advances the watermark, then the data
from all other vehicles will be considered late, because Watermarks are
global.
Given my problem, is there a way I can implement different watermarks for
different keys? If not directly possible, is there some way I can simulate
it to suit my application?
Any help would be greatly appreciated!
--
Sent from:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Locked
