Why would indefinitely growing state an issue for Flink while doing stream to stream joins?

Posted by kant kodali on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Why-would-indefinitely-growing-state-an-issue-for-Flink-while-doing-stream-to-stream-joins-tp32188.html

Hi All,

The doc https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/joins.html#regular-joins says the following. 

"However, this operation has an important implication: it requires to keep both sides of the join input in Flink’s state forever. Thus, the resource usage will grow indefinitely as well, if one or both input tables are continuously growing"

I wonder why this would be an issue especially when the state is stored in RocksDB which in turn is backed by disk? 

I have a use case where I might need to do stream-stream join or some emulation of that across say 6 or more tables and I don't know for sure how long I need to keep the state because a row today can join with a row a year or two years from now. will that be an issue? do I need to think about designing a solution in another way without using stream-stream join?

Thanks!