Flink join stream where one stream is coming 5 minutes late

[Abhijeet Kumar]

Hello Team,

I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know. 

Thanks,

Abhijeet Kumar Software Development Engineer, Sentienz Solutions Pvt Ltd Cognitive Data Platform - Perceive the Data ! [hidden email] |www.sentienz.com | Bengaluru

[Taher Koitawala]

May I ask why you want to have 2 differences between window time? What's the use case?

On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <[hidden email] wrote:

Hello Team,

I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know. 

Thanks,

Abhijeet Kumar Software Development Engineer, Sentienz Solutions Pvt Ltd Cognitive Data Platform - Perceive the Data ! [hidden email] |www.sentienz.com | Bengaluru

[Abhijeet Kumar]

I already mentioned above, one is coming late by 5 mins. Maybe my approach isn't correct, and I asked to correct me if I'm wrong. 

On 26-Nov-2018, at 5:56 PM, Taher Koitawala <[hidden email]> wrote:

May I ask why you want to have 2 differences between window time? What's the use case?

On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <[hidden email] wrote:

Hello Team,

I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know. 

Thanks,

Abhijeet Kumar Software Development Engineer, Sentienz Solutions Pvt Ltd Cognitive Data Platform - Perceive the Data ! [hidden email] |www.sentienz.com | Bengaluru

[Xingcan Cui]

Hi Abhijeet,

If you want to perform window-join in the DataStream API, the window configurations on both sides must be exactly the same.

For your case, maybe you can try adding a 5 mins delay on event times (and watermarks) of the faster stream. 

Hope that helps.

Best,

Xingcan

On Nov 26, 2018, at 12:44 PM, Abhijeet Kumar <[hidden email]> wrote:

I already mentioned above, one is coming late by 5 mins. Maybe my approach isn't correct, and I asked to correct me if I'm wrong. 

On 26-Nov-2018, at 5:56 PM, Taher Koitawala <[hidden email]> wrote:

May I ask why you want to have 2 differences between window time? What's the use case?

On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <[hidden email] wrote:

Hello Team,

I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know. 

Thanks,

Abhijeet Kumar Software Development Engineer, Sentienz Solutions Pvt Ltd Cognitive Data Platform - Perceive the Data ! [hidden email] |www.sentienz.com | Bengaluru

[Abhijeet Kumar]

Thanks for the response, Xingcan. I got your point and the whole idea.

On 27-Nov-2018, at 12:15 AM, Xingcan Cui <[hidden email]> wrote:

Hi Abhijeet,

If you want to perform window-join in the DataStream API, the window configurations on both sides must be exactly the same.

For your case, maybe you can try adding a 5 mins delay on event times (and watermarks) of the faster stream. 

Hope that helps.

Best,

Xingcan

On Nov 26, 2018, at 12:44 PM, Abhijeet Kumar <[hidden email]> wrote:

I already mentioned above, one is coming late by 5 mins. Maybe my approach isn't correct, and I asked to correct me if I'm wrong. 

On 26-Nov-2018, at 5:56 PM, Taher Koitawala <[hidden email]> wrote:

May I ask why you want to have 2 differences between window time? What's the use case?

On Mon 26 Nov, 2018, 5:53 PM Abhijeet Kumar <[hidden email] wrote:

Hello Team,

I've to join two stream where one stream is coming late. So, I planned doing it by creating two windows, for first window the size will be 5 minutes and for the other it will be 10 minutes. Then, I'll perform the join operation. Is my solution correct because I don't know whether one window will sync with other or not because one is getting finished earlier and another is late. If there is a better way to achieve the same, please let me know. 

Thanks,

Abhijeet Kumar Software Development Engineer, Sentienz Solutions Pvt Ltd Cognitive Data Platform - Perceive the Data ! [hidden email] |www.sentienz.com | Bengaluru