FLIP-17: Side Inputs

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

FLIP-17: Side Inputs

Elias Levy
A bit late to this discussion, but I wanted to reiterate something that others also said. Side input readiness, and blocking until that is the case, is an important feature.  This is specially true when the side input is used as a configuration stream.  You don't want the main stream to be processed until at least the minimal required configuration is loaded.  

I'll also note that Kafka Streams has struggled with the same problem (KAFKA-4113), but it has the advantage that KS prioritizes consumption from sources based on timestamp.  So with KS if your KTable config records have an earlier timestamp than the stream records they are joined with, they will be consumed first (although it does so on a best effort basis).