About Sliding window

[Zhangrucong]

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

 

[Kostas Kloudas]

Hi Zhangrucong,

Sliding windows only support time-based slide. 

So your use-case is not supported out-of-the-box.

But, if you describe a bit more what you want to do, 

we may be able to find a way together to do your job using 

the currently offered functionality.

Kostas

On Oct 11, 2016, at 1:20 PM, Zhangrucong <[hidden email]> wrote:

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

[Zhangrucong]

Hi Kostas:

Thank you for your rapid response!

 

My use-case is that :

For every incoming event, we want to age the out-of-date event , count the event in window and send the result.

 

For example:

The events are coming as flowing:

 

We want flowing result:

 

 

By the way, In StreamSQL API, in FILP11, It will realize row window. It seems that the function of Slide Event-time row-window suits my use-case. Does data stream API  support row window?

 

Thanks !

 

发件人: Kostas Kloudas [mailto:[hidden email]]
发送时间: 2016年10月11日 19:38
收件人: [hidden email]
主题: Re: About Sliding window

 

Hi Zhangrucong,

 

Sliding windows only support time-based slide. 

So your use-case is not supported out-of-the-box.

 

But, if you describe a bit more what you want to do, 

we may be able to find a way together to do your job using 

the currently offered functionality.

 

Kostas

 

On Oct 11, 2016, at 1:20 PM, Zhangrucong <[hidden email]> wrote:

 

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

 

[Kostas Kloudas]

Hello again,

Sorry for the delay but I cannot really understand your use case.

Could you explain a bit more what do you mean by “out-of-date” event and “aging” an event?

Also your windows are of a certain duration or global?

Thanks,

Kostas

On Oct 11, 2016, at 3:04 PM, Zhangrucong <[hidden email]> wrote:

Hi Kostas:

Thank you for your rapid response!

 

My use-case is that :

For every incoming event, we want to age the out-of-date event , count the event in window and send the result.

 

For example:

The events are coming as flowing:

<image002.png>

 

We want flowing result:

<image004.png>

 

 

By the way, In StreamSQL API, in FILP11, It will realize row window. It seems that the function of Slide Event-time row-window suits my use-case. Does data stream API  support row window?

 

Thanks !

 

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月11日 19:38
收件人: [hidden email]
主题: Re: About Sliding window

 

Hi Zhangrucong,

 

Sliding windows only support time-based slide. 

So your use-case is not supported out-of-the-box.

 

But, if you describe a bit more what you want to do, 

we may be able to find a way together to do your job using 

the currently offered functionality.

 

Kostas

 

On Oct 11, 2016, at 1:20 PM, Zhangrucong <[hidden email]> wrote:

 

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

 

<oledata.mso>

[Zhangrucong]

Hi Kostas:

            It doesn’t matter. Can you see the picture? My user case is:

 

1、The events are coming according to the following order

At 9:01 e1 is coming

At 9:02 e2 is coming

At 9:06  e3 is coming

At 9:08   e4 is coming

 

The time is system time.

 

2、And  event aging time is 5 minutes.

 

3、

       At 9:01 e1 is coming, aged nothing, store e1,we count e1 and send the result.

       At 9:02 e2 is coming,  aged nothing, store e2,  We count e1 and e2. and send the result.

      At 9:06  e3 is coming,  aged e1,  store e3, we count e2 and e3, and send the result.

     At 9:08   e4 is coming,  aged e2,  store e4, we count e3 and e4, and send the result.

 

 

I think I need a certain duration window.

 

Thank you very much!

发件人: Kostas Kloudas [mailto:[hidden email]]
发送时间: 2016年10月12日 21:11
收件人: Zhangrucong
抄送: [hidden email]
主题: Re: About Sliding window

 

Hello again,

 

Sorry for the delay but I cannot really understand your use case.

Could you explain a bit more what do you mean by “out-of-date” event and “aging” an event?

 

Also your windows are of a certain duration or global?

 

Thanks,

Kostas

 

On Oct 11, 2016, at 3:04 PM, Zhangrucong <[hidden email]> wrote:

 

Hi Kostas:

Thank you for your rapid response!

 

My use-case is that :

For every incoming event, we want to age the out-of-date event , count the event in window and send the result.

 

For example:

The events are coming as flowing:

<image002.png>

 

We want flowing result:

<image004.png>

 

 

By the way, In StreamSQL API, in FILP11, It will realize row window. It seems that the function of Slide Event-time row-window suits my use-case. Does data stream API  support row window?

 

Thanks !

 

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月11日 19:38
收件人: [hidden email]
主题: Re: About Sliding window

 

Hi Zhangrucong,

 

Sliding windows only support time-based slide. 

So your use-case is not supported out-of-the-box.

 

But, if you describe a bit more what you want to do, 

we may be able to find a way together to do your job using 

the currently offered functionality.

 

Kostas

 

On Oct 11, 2016, at 1:20 PM, Zhangrucong <[hidden email]> wrote:

 

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

 

<oledata.mso>

 

[Kostas Kloudas]

Hello,

So in your previous figure (yesterday) when e3 arrives, also e2 should be included in the result, right?

In this case, I think that what you need is a Session window with gap equal to your event aging duration and

an evictor that evicts the elements that lag behind more than the gap duration.

The latter, the evictor that I am describing, is not currently supported in Flink but there is an ongoing 

discussion in the dev mailing list about it. So it is worth having a look there and participate in the discussion.

I also loop in Aljoscha in the discussion, in case he has another solution that you can deploy right-away.

Thanks,

Kostas

On Oct 12, 2016, at 3:36 PM, Zhangrucong <[hidden email]> wrote:

Hi Kostas:

            It doesn’t matter. Can you see the picture? My user case is:

 

1、The events are coming according to the following order

<image004.png>

At 9:01 e1 is coming

At 9:02 e2 is coming

At 9:06  e3 is coming

At 9:08   e4 is coming

 

The time is system time.

 

2、And  event aging time is 5 minutes.

 

3、

       At 9:01 e1 is coming, aged nothing, store e1,we count e1 and send the result.

       At 9:02 e2 is coming,  aged nothing, store e2,  We count e1 and e2. and send the result.

      At 9:06  e3 is coming,  aged e1,  store e3, we count e2 and e3, and send the result.

     At 9:08   e4 is coming,  aged e2,  store e4, we count e3 and e4, and send the result.

 

 

I think I need a certain duration window.

 

Thank you very much!

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月12日 21:11
收件人: Zhangrucong
抄送: [hidden email]
主题: Re: About Sliding window

 

Hello again,

 

Sorry for the delay but I cannot really understand your use case.

Could you explain a bit more what do you mean by “out-of-date” event and “aging” an event?

 

Also your windows are of a certain duration or global?

 

Thanks,

Kostas

 

On Oct 11, 2016, at 3:04 PM, Zhangrucong <[hidden email]> wrote:

 

Hi Kostas:

Thank you for your rapid response!

 

My use-case is that :

For every incoming event, we want to age the out-of-date event , count the event in window and send the result.

 

For example:

The events are coming as flowing:

<image002.png>

 

We want flowing result:

<image004.png>

 

 

By the way, In StreamSQL API, in FILP11, It will realize row window. It seems that the function of Slide Event-time row-window suits my use-case. Does data stream API  support row window?

 

Thanks !

 

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月11日 19:38
收件人: [hidden email]
主题: Re: About Sliding window

 

Hi Zhangrucong,

 

Sliding windows only support time-based slide. 

So your use-case is not supported out-of-the-box.

 

But, if you describe a bit more what you want to do, 

we may be able to find a way together to do your job using 

the currently offered functionality.

 

Kostas

 

On Oct 11, 2016, at 1:20 PM, Zhangrucong <[hidden email]> wrote:

 

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

 

<oledata.mso>

 

<oledata.mso>

[Zhangrucong]

Hi Kostas：

Thanks for your answer.

 

So in your previous figure (yesterday) when e3 arrives, also e2 should be included in the result, right?

------zhangrucong: In Oct 11 email, e2 is coming at 9:02, e3 is coming at 9:07, and the aging time is 5 mins. So When e3 coming, e2 is aged. E2 is not in the result!

 

In the mail, you say you have discussion. Can you show me the link , I want to take part in it.

 

Best wishes!

 

发件人: Kostas Kloudas [mailto:[hidden email]]
发送时间: 2016年10月12日 22:32
收件人: Zhangrucong
抄送: [hidden email]; Aljoscha Krettek
主题: Re: About Sliding window

 

Hello,

 

So in your previous figure (yesterday) when e3 arrives, also e2 should be included in the result, right?

 

In this case, I think that what you need is a Session window with gap equal to your event aging duration and

an evictor that evicts the elements that lag behind more than the gap duration.

 

The latter, the evictor that I am describing, is not currently supported in Flink but there is an ongoing 

discussion in the dev mailing list about it. So it is worth having a look there and participate in the discussion.

 

I also loop in Aljoscha in the discussion, in case he has another solution that you can deploy right-away.

 

Thanks,

Kostas

 

On Oct 12, 2016, at 3:36 PM, Zhangrucong <[hidden email]> wrote:

 

Hi Kostas:

            It doesn’t matter. Can you see the picture? My user case is:

 

1、The events are coming according to the following order

<image004.png>

At 9:01 e1 is coming

At 9:02 e2 is coming

At 9:06  e3 is coming

At 9:08   e4 is coming

 

The time is system time.

 

2、And  event aging time is 5 minutes.

 

3、

       At 9:01 e1 is coming, aged nothing, store e1,we count e1 and send the result.

       At 9:02 e2 is coming,  aged nothing, store e2,  We count e1 and e2. and send the result.

      At 9:06  e3 is coming,  aged e1,  store e3, we count e2 and e3, and send the result.

     At 9:08   e4 is coming,  aged e2,  store e4, we count e3 and e4, and send the result.

 

 

I think I need a certain duration window.

 

Thank you very much!

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月12日 21:11
收件人: Zhangrucong
抄送: [hidden email]
主题: Re: About Sliding window

 

Hello again,

 

Sorry for the delay but I cannot really understand your use case.

Could you explain a bit more what do you mean by “out-of-date” event and “aging” an event?

 

Also your windows are of a certain duration or global?

 

Thanks,

Kostas

 

On Oct 11, 2016, at 3:04 PM, Zhangrucong <[hidden email]> wrote:

 

Hi Kostas:

Thank you for your rapid response!

 

My use-case is that :

For every incoming event, we want to age the out-of-date event , count the event in window and send the result.

 

For example:

The events are coming as flowing:

<image002.png>

 

We want flowing result:

<image004.png>

 

 

By the way, In StreamSQL API, in FILP11, It will realize row window. It seems that the function of Slide Event-time row-window suits my use-case. Does data stream API  support row window?

 

Thanks !

 

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月11日 19:38
收件人: [hidden email]
主题: Re: About Sliding window

 

Hi Zhangrucong,

 

Sliding windows only support time-based slide. 

So your use-case is not supported out-of-the-box.

 

But, if you describe a bit more what you want to do, 

we may be able to find a way together to do your job using 

the currently offered functionality.

 

Kostas

 

On Oct 11, 2016, at 1:20 PM, Zhangrucong <[hidden email]> wrote:

 

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

 

<oledata.mso>

 

<oledata.mso>

 

[Kostas Kloudas]

Hi Zhangrucong,

The relevant document is Flip-4: https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor

There you can also find a link to the discussion.

Cheers,

Kostas

On Oct 13, 2016, at 4:17 AM, Zhangrucong <[hidden email]> wrote:

Hi Kostas：

Thanks for your answer.

 

So in your previous figure (yesterday) when e3 arrives, also e2 should be included in the result, right?

------zhangrucong: In Oct 11 email, e2 is coming at 9:02, e3 is coming at 9:07, and the aging time is 5 mins. So When e3 coming, e2 is aged. E2 is not in the result! 

 

In the mail, you say you have discussion. Can you show me the link , I want to take part in it.

 

Best wishes!

 

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月12日 22:32
收件人: Zhangrucong
抄送: [hidden email]; Aljoscha Krettek
主题: Re: About Sliding window

 

Hello,

 

So in your previous figure (yesterday) when e3 arrives, also e2 should be included in the result, right?

 

In this case, I think that what you need is a Session window with gap equal to your event aging duration and

an evictor that evicts the elements that lag behind more than the gap duration.

 

The latter, the evictor that I am describing, is not currently supported in Flink but there is an ongoing 

discussion in the dev mailing list about it. So it is worth having a look there and participate in the discussion.

 

I also loop in Aljoscha in the discussion, in case he has another solution that you can deploy right-away.

 

Thanks,

Kostas

 

On Oct 12, 2016, at 3:36 PM, Zhangrucong <[hidden email]> wrote:

 

Hi Kostas:

            It doesn’t matter. Can you see the picture? My user case is:

 

1、The events are coming according to the following order

<image004.png>

At 9:01 e1 is coming

At 9:02 e2 is coming

At 9:06  e3 is coming

At 9:08   e4 is coming

 

The time is system time.

 

2、And  event aging time is 5 minutes.

 

3、

       At 9:01 e1 is coming, aged nothing, store e1,we count e1 and send the result.

       At 9:02 e2 is coming,  aged nothing, store e2,  We count e1 and e2. and send the result.

      At 9:06  e3 is coming,  aged e1,  store e3, we count e2 and e3, and send the result.

     At 9:08   e4 is coming,  aged e2,  store e4, we count e3 and e4, and send the result.

 

 

I think I need a certain duration window.

 

Thank you very much!

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月12日 21:11
收件人: Zhangrucong
抄送: [hidden email]
主题: Re: About Sliding window

 

Hello again,

 

Sorry for the delay but I cannot really understand your use case.

Could you explain a bit more what do you mean by “out-of-date” event and “aging” an event?

 

Also your windows are of a certain duration or global?

 

Thanks,

Kostas

 

On Oct 11, 2016, at 3:04 PM, Zhangrucong <[hidden email]> wrote:

 

Hi Kostas:

Thank you for your rapid response!

 

My use-case is that :

For every incoming event, we want to age the out-of-date event , count the event in window and send the result.

 

For example:

The events are coming as flowing:

<image002.png>

 

We want flowing result:

<image004.png>

 

 

By the way, In StreamSQL API, in FILP11, It will realize row window. It seems that the function of Slide Event-time row-window suits my use-case. Does data stream API  support row window?

 

Thanks !

 

发件人: Kostas Kloudas [[hidden email]] 
发送时间: 2016年10月11日 19:38
收件人: [hidden email]
主题: Re: About Sliding window

 

Hi Zhangrucong,

 

Sliding windows only support time-based slide. 

So your use-case is not supported out-of-the-box.

 

But, if you describe a bit more what you want to do, 

we may be able to find a way together to do your job using 

the currently offered functionality.

 

Kostas

 

On Oct 11, 2016, at 1:20 PM, Zhangrucong <[hidden email]> wrote:

 

Hello everyone:

  Now, I am want to use DataStream sliding window API. I look at the API and I have a question, dose the sliding time window support sliding by every incoming event?

 

Thanks in advance!

 

<oledata.mso>

 

<oledata.mso>