Watermark and multiple streams

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Watermark and multiple streams

Filipe Couto-2

Hello.

 

I’m joining several data streams, using ConnectedStreams. Let’s say something like A connect B which outputs AB, and then I join AB with C, which outputs ABC.

 

However, the relationship between A and B, or AB and C may be of 1 to many, or 1 to 1, depending on the case. For the 1 to 1, it’s expected to produce an output as soon as I obtain both records that match the same key, but for the 1 to many, it’s not so simple, there’s no event that guarantees that I have obtained all of my «many» records. For this, I’ve searched and found out that I have to implement an onTimer method, in a Process Function. I also read that an event timer is registered in a queue and when a new watermark arrives, the event timers that «passed» will be triggered.

 

Given this, I’m running into trouble when generating new watermarks like in the example: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/event_timestamps_watermarks.html#with-periodic-watermarks

 

The solution I’ve reached basically processes (on the onTimer method) all the records that I obtained previously when the first watermark arrives, but after that, it stops triggering.

 

I read that watermarks are global, so how can I create a new watermark when working with several streams?

 

Thank you

Reply | Threaded
Open this post in threaded view
|

Re: Watermark and multiple streams

Tzu-Li (Gordon) Tai
Hi,

How are your registering your event time timers on processElement?
If you are continuously registering them, and watermarks are correctly
generated upstream, then the onTimer method should be invoked properly.

For your 1-to-many case, I would assume that whenever a new key arrives
(that previously has not seen events with the same key from other streams),
an event time timer is registered to be fired after a certain amount of time
which you allow to wait for other matching join records.

Does this help?

Cheers,
Gordon



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/