Question about Window Tigger

[Chang Liu]

Dear All,

I have a question about the Window Trigger: let’s say i would like like use the SlidingEventTimeWindow (60 seconds window size + 1 second window shift) to count the number of records per window. And I am using Event Time with periodic watermarking with certain maxOurOfOrderness time.

Sometimes, what happens is: during certain time, there is no incoming event, and then the watermark for triggering the window fire is not coming.  Then, the last several records will be just stayed in the window.  It will fire only when the window sees the watermark to trigger.

What I would like to achieve is: if there is no just watermark coming within certain time (maybe this time is system clock time?), I can still trigger the window to fire no matter whether there is new event coming or not. Then I can still get the window count for this window, without waiting the next event, which could be coming after a long time.

Do you have any idea how can I do this? Many Thanks :)

Best regards/祝好,

Chang Liu 刘畅

[Rong Rong]

Hi Chang,

There were some previous discussion regarding how to debug watermark and window triggers[1].

Basically if there's no data for some partitions there's no way to advance watermark. As it would not be able to determine whether this is due to network failure or actually there's no data arriving at the source.

I think your use case is better of using SlidingProcessingTimeWindow.

Thanks,

Rong

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/watermark-does-not-progress-td22315.html

On Wed, Sep 19, 2018 at 1:48 AM Chang Liu <[hidden email]> wrote:

Dear All,

I have a question about the Window Trigger: let’s say i would like like use the SlidingEventTimeWindow (60 seconds window size + 1 second window shift) to count the number of records per window. And I am using Event Time with periodic watermarking with certain maxOurOfOrderness time.

Sometimes, what happens is: during certain time, there is no incoming event, and then the watermark for triggering the window fire is not coming.  Then, the last several records will be just stayed in the window.  It will fire only when the window sees the watermark to trigger.

What I would like to achieve is: if there is no just watermark coming within certain time (maybe this time is system clock time?), I can still trigger the window to fire no matter whether there is new event coming or not. Then I can still get the window count for this window, without waiting the next event, which could be coming after a long time.

Do you have any idea how can I do this? Many Thanks :)

Best regards/祝好,

Chang Liu 刘畅

[Chang Liu]

Hi Rong,

Thanks for your reply. :)

Best regards/祝好,

Chang Liu 刘畅

On 19 Sep 2018, at 18:20, Rong Rong <[hidden email]> wrote:

Hi Chang,

There were some previous discussion regarding how to debug watermark and window triggers[1].

Basically if there's no data for some partitions there's no way to advance watermark. As it would not be able to determine whether this is due to network failure or actually there's no data arriving at the source.

I think your use case is better of using SlidingProcessingTimeWindow.

Thanks,

Rong

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/watermark-does-not-progress-td22315.html

On Wed, Sep 19, 2018 at 1:48 AM Chang Liu <[hidden email]> wrote:

Dear All,

I have a question about the Window Trigger: let’s say i would like like use the SlidingEventTimeWindow (60 seconds window size + 1 second window shift) to count the number of records per window. And I am using Event Time with periodic watermarking with certain maxOurOfOrderness time.

Sometimes, what happens is: during certain time, there is no incoming event, and then the watermark for triggering the window fire is not coming.  Then, the last several records will be just stayed in the window.  It will fire only when the window sees the watermark to trigger.

What I would like to achieve is: if there is no just watermark coming within certain time (maybe this time is system clock time?), I can still trigger the window to fire no matter whether there is new event coming or not. Then I can still get the window count for this window, without waiting the next event, which could be coming after a long time.

Do you have any idea how can I do this? Many Thanks :)

Best regards/祝好,

Chang Liu 刘畅

[Till Rohrmann]

Hi Chang Liu,

maybe you could use the AssignerWithPeriodicWatermarks to build a custom watermark assigner which creates the watermarks based on the incoming events and if it detects that no events are coming, that it progresses the watermark with respect to the wall clock time.

Cheers,

Till

On Tue, Sep 25, 2018 at 10:36 AM Chang Liu <[hidden email]> wrote:

Hi Rong,

Thanks for your reply. :)

Best regards/祝好,

Chang Liu 刘畅

On 19 Sep 2018, at 18:20, Rong Rong <[hidden email]> wrote:

Hi Chang,

There were some previous discussion regarding how to debug watermark and window triggers[1].

Basically if there's no data for some partitions there's no way to advance watermark. As it would not be able to determine whether this is due to network failure or actually there's no data arriving at the source.

I think your use case is better of using SlidingProcessingTimeWindow.

Thanks,

Rong

[1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/watermark-does-not-progress-td22315.html

On Wed, Sep 19, 2018 at 1:48 AM Chang Liu <[hidden email]> wrote:

Dear All,

I have a question about the Window Trigger: let’s say i would like like use the SlidingEventTimeWindow (60 seconds window size + 1 second window shift) to count the number of records per window. And I am using Event Time with periodic watermarking with certain maxOurOfOrderness time.

Sometimes, what happens is: during certain time, there is no incoming event, and then the watermark for triggering the window fire is not coming.  Then, the last several records will be just stayed in the window.  It will fire only when the window sees the watermark to trigger.

What I would like to achieve is: if there is no just watermark coming within certain time (maybe this time is system clock time?), I can still trigger the window to fire no matter whether there is new event coming or not. Then I can still get the window count for this window, without waiting the next event, which could be coming after a long time.

Do you have any idea how can I do this? Many Thanks :)

Best regards/祝好,

Chang Liu 刘畅

[Chang Liu]

Hi Till,

You mean use AssignerWithPeriodicWatermarks but combine the logic of Event time and Processing Time (or to say depend on the time difference/interval of Processing time, but the Watermark value is still depending on the Event time right)?

One additional question: where do I config this: ExecutionConfig.setAutoWatermarkInterval(…) ?

Many thanks :)

Best regards/祝好,

Chang Liu 刘畅

On 25 Sep 2018, at 23:41, Till Rohrmann <[hidden email]> wrote:

AssignerWithPeriodicWatermarks

[Till Rohrmann]

You configure the ExecutionConfig#setAutoWatermarkInterval on the StreamExecutionEnvironment by calling getConfig().

Cheers,

Till

On Tue, Sep 25, 2018 at 11:54 PM Chang Liu <[hidden email]> wrote:

Hi Till,

You mean use AssignerWithPeriodicWatermarks but combine the logic of Event time and Processing Time (or to say depend on the time difference/interval of Processing time, but the Watermark value is still depending on the Event time right)?

One additional question: where do I config this: ExecutionConfig.setAutoWatermarkInterval(…) ?

Many thanks :)

Best regards/祝好,

Chang Liu 刘畅

On 25 Sep 2018, at 23:41, Till Rohrmann <[hidden email]> wrote:

AssignerWithPeriodicWatermarks