Does Flink support joining multiple streams based on event time window now?

[Yifei Li]

Hi,

I am new to Flink and I've read some documentation and think Flink may fit my scenario.

Here is my scenario:

1. Assume I have 3 streams: S1(id, name, email, action, date), S2(id, name, email, level, date), S3(id, name, position, date).

2. S2 always delays(hours to days, not determined..) 

3. Based on the event time, I want to join S1, S2 and S3 every 5 minutes. The join is like a SQL join:

    select S1.name, S3.position from S1, S2, S3 where S1.id = S2.id and S1.id = S3.id and S1.action = 'download' and S2.level = 5

Can I use Flink for my scenario? Is yes, can anyone point me to some working examples(I found some examples but they are outdated), or tell me some workaround to solve this problem? If no, can anyone tell me the reasons?

Thanks,

Yifei 

 

[Aljoscha Krettek]

Hi,

right now, there is no built-in support for n-ary joins. I am working on this, however.

For now you can simulate n-ary joins by using a tagged union and doing the join yourself in a WindowFunction. I created a small example that demonstrates this: https://gist.github.com/aljoscha/a2a213d90c7c1bc67e71fabaa82fba4a

I hope this helps, and please let us know if you want to know more.

Cheers,

Aljoscha

On Tue, 19 Apr 2016 at 02:11 Yifei Li <[hidden email]> wrote:

Hi,

I am new to Flink and I've read some documentation and think Flink may fit my scenario.

Here is my scenario:

1. Assume I have 3 streams: S1(id, name, email, action, date), S2(id, name, email, level, date), S3(id, name, position, date).

2. S2 always delays(hours to days, not determined..) 

3. Based on the event time, I want to join S1, S2 and S3 every 5 minutes. The join is like a SQL join:

    select S1.name, S3.position from S1, S2, S3 where S1.id = S2.id and S1.id = S3.id and S1.action = 'download' and S2.level = 5

Can I use Flink for my scenario? Is yes, can anyone point me to some working examples(I found some examples but they are outdated), or tell me some workaround to solve this problem? If no, can anyone tell me the reasons?

Thanks,

Yifei 

 

[Till Rohrmann]

Hi Yifei,

if you don't wanna implement your own join operator, then you could also chain two join operations. I created a small example to demonstrate that: https://gist.github.com/tillrohrmann/c074b4eedb9deaf9c8ca2a5e124800f3. However, bare in mind that for this approach you will construct two windows which might be a bit more costly than Aljoscha's approach.

Cheers,

Till

On Tue, Apr 19, 2016 at 11:32 AM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

right now, there is no built-in support for n-ary joins. I am working on this, however.

For now you can simulate n-ary joins by using a tagged union and doing the join yourself in a WindowFunction. I created a small example that demonstrates this: https://gist.github.com/aljoscha/a2a213d90c7c1bc67e71fabaa82fba4a

I hope this helps, and please let us know if you want to know more.

Cheers,

Aljoscha

On Tue, 19 Apr 2016 at 02:11 Yifei Li <[hidden email]> wrote:

Hi,

I am new to Flink and I've read some documentation and think Flink may fit my scenario.

Here is my scenario:

1. Assume I have 3 streams: S1(id, name, email, action, date), S2(id, name, email, level, date), S3(id, name, position, date).

2. S2 always delays(hours to days, not determined..) 

3. Based on the event time, I want to join S1, S2 and S3 every 5 minutes. The join is like a SQL join:

    select S1.name, S3.position from S1, S2, S3 where S1.id = S2.id and S1.id = S3.id and S1.action = 'download' and S2.level = 5

Can I use Flink for my scenario? Is yes, can anyone point me to some working examples(I found some examples but they are outdated), or tell me some workaround to solve this problem? If no, can anyone tell me the reasons?

Thanks,

Yifei 

 

[Yifei Li]

Hi Till and Aljoscha,

Thank you so much for your suggestions and I'll try them out. I have another question.

Since S2 my be days delayed, so there are may be lots of windows and large amount of data stored in memory waiting for computation. How does Flink deal with that？

Thanks,

Yifei 

On Tue, Apr 19, 2016 at 2:55 AM, Till Rohrmann <[hidden email]> wrote:

Hi Yifei,

if you don't wanna implement your own join operator, then you could also chain two join operations. I created a small example to demonstrate that: https://gist.github.com/tillrohrmann/c074b4eedb9deaf9c8ca2a5e124800f3. However, bare in mind that for this approach you will construct two windows which might be a bit more costly than Aljoscha's approach.

Cheers,

Till

On Tue, Apr 19, 2016 at 11:32 AM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

right now, there is no built-in support for n-ary joins. I am working on this, however.

For now you can simulate n-ary joins by using a tagged union and doing the join yourself in a WindowFunction. I created a small example that demonstrates this: https://gist.github.com/aljoscha/a2a213d90c7c1bc67e71fabaa82fba4a

I hope this helps, and please let us know if you want to know more.

Cheers,

Aljoscha

On Tue, 19 Apr 2016 at 02:11 Yifei Li <[hidden email]> wrote:

Hi,

I am new to Flink and I've read some documentation and think Flink may fit my scenario.

Here is my scenario:

1. Assume I have 3 streams: S1(id, name, email, action, date), S2(id, name, email, level, date), S3(id, name, position, date).

2. S2 always delays(hours to days, not determined..) 

3. Based on the event time, I want to join S1, S2 and S3 every 5 minutes. The join is like a SQL join:

    select S1.name, S3.position from S1, S2, S3 where S1.id = S2.id and S1.id = S3.id and S1.action = 'download' and S2.level = 5

Can I use Flink for my scenario? Is yes, can anyone point me to some working examples(I found some examples but they are outdated), or tell me some workaround to solve this problem? If no, can anyone tell me the reasons?

Thanks,

Yifei 

 

[Till Rohrmann]

If the data exceeds the main memory of your machine, then you should use the RocksDBStateBackend as a state backend. It allows you to store state (including windows) on disk. Thus, the size of state you can store is then limited by your hard disk capacity.

If the expected data size can be kept in memory, then you should use the FileStateBackend which keeps your state data in memory and only writes it to disk if you draw a checkpoint.

Cheers,
Till

​

On Tue, Apr 19, 2016 at 6:08 PM, Yifei Li <[hidden email]> wrote:

Hi Till and Aljoscha,

Thank you so much for your suggestions and I'll try them out. I have another question.

Since S2 my be days delayed, so there are may be lots of windows and large amount of data stored in memory waiting for computation. How does Flink deal with that？

Thanks,

Yifei 

On Tue, Apr 19, 2016 at 2:55 AM, Till Rohrmann <[hidden email]> wrote:

Hi Yifei,

if you don't wanna implement your own join operator, then you could also chain two join operations. I created a small example to demonstrate that: https://gist.github.com/tillrohrmann/c074b4eedb9deaf9c8ca2a5e124800f3. However, bare in mind that for this approach you will construct two windows which might be a bit more costly than Aljoscha's approach.

Cheers,

Till

On Tue, Apr 19, 2016 at 11:32 AM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

right now, there is no built-in support for n-ary joins. I am working on this, however.

For now you can simulate n-ary joins by using a tagged union and doing the join yourself in a WindowFunction. I created a small example that demonstrates this: https://gist.github.com/aljoscha/a2a213d90c7c1bc67e71fabaa82fba4a

I hope this helps, and please let us know if you want to know more.

Cheers,

Aljoscha

On Tue, 19 Apr 2016 at 02:11 Yifei Li <[hidden email]> wrote:

Hi,

I am new to Flink and I've read some documentation and think Flink may fit my scenario.

Here is my scenario:

1. Assume I have 3 streams: S1(id, name, email, action, date), S2(id, name, email, level, date), S3(id, name, position, date).

2. S2 always delays(hours to days, not determined..) 

3. Based on the event time, I want to join S1, S2 and S3 every 5 minutes. The join is like a SQL join:

    select S1.name, S3.position from S1, S2, S3 where S1.id = S2.id and S1.id = S3.id and S1.action = 'download' and S2.level = 5

Can I use Flink for my scenario? Is yes, can anyone point me to some working examples(I found some examples but they are outdated), or tell me some workaround to solve this problem? If no, can anyone tell me the reasons?

Thanks,

Yifei