|
|
Hi all, I have a streaming job that:
reads from kafka --> maps events to some other DataStream --> keyby(0) --> reduces a timewindow of 15 seconds processing time (probably where the spills to disk happen?) and writes back to a redis sink.
When starting up, everything works great. The problem is, that after a while, the disk space get's full by flinks spilling data to disk (probably does not fit into memory, even though I increased taskmanager.heap.mb to 4096M) and the reading speed is severely slowed down, still haven't figured out why, because, when restarting, the gap from kafka is closed really quick.
My question is, is that temp data supposed to ever be cleaned/deleted while the flink job is running?, could not find any resources on this.
thanks!
|
Locked
