Re: AW: Is watermark used by joining two streams

Posted by Fabian Hueske-2 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Is-watermark-used-by-joining-two-streams-tp14513p14547.html

Hi,

@Wei: You can implement very different behavior using a CoProcessFunction. However, if your operator is time-based, the logical time of the operator will be the minimum time of both streams (time of the "slower" watermark).

@Vijay: I did not understand what your requirements are. Do you want to join or merge streams? Those are two different things. This thread discusses joins not merging.

Best,

Fabian

2017-07-31 4:24 GMT+02:00 G.S.Vijay Raajaa <[hidden email]>:

Hi Fabian,

How do I order by the merge time. Let's say I merge the stream at T1. I wanted to drop T2 merge if T2 < T1. Now depending on the arrival of data from individual stream and the time at which the merge happens, they become out of order. Any thoughts will be really appreciated.

Regards,

Vijay Raajaa GS

On Jul 31, 2017 1:14 AM, "wei" <[hidden email]> wrote:

Hello Fabian,

 

thank you for your answer!

 

Does it mean that the operator will wait until get two watermarks from the input streams and emits then the “slower” watermark?

 

Best regards

Wei

 

Von: Fabian Hueske [mailto:[hidden email]]
Gesendet: Sunday, July 30, 2017 11:17 AM
An: xie wei
Cc: user
Betreff: Re: Is watermark used by joining two streams

 

Periodic and punctuated watermarks only differ in the way that they are generated. Afterwards they are treated the same.

An operator with two input streams will always sync its own watermarks to the watermarks of both input streams, i.e., to the "slower" watermark of both inputs.

So if the left input says it is 12:14 and the right says it is 11:53, the operator will have a internal time of 11:53 and emit watermarks according to that time.

Hope that helps,

Fabian

 

 

2017-07-28 15:00 GMT+02:00 xie wei <[hidden email]>:

Hello,

i want to join two streams based on event time window, every stream has its own watermark, one has priodic watermark and the other has punctuated watermark.

are the watermarks used to trigger the join? if yes, which one and how is it used?

Thank you and best regards

Wei