Manual trigger the window in fold operator or incremental aggregation

[zhen li]

Hi all:
    How can I trigger the window manually in  fold operator or incremental aggregation? For example, when some conditions is meet，althouth the count window or time window is not reach

[Niels van Kaam]

Hi Zhen Li,

You can control when a windowed stream emits data with "Triggers". See: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#triggers 

Flink comes with a couple of default triggers, but you can also create your own by implementing https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/windowing/triggers/Trigger.html.

Note that this does not change the window, but just causes the windowedstream to emit intermediate results to the next operator.

Does this answer your question?

Cheers,

Niels

On Wed, Oct 17, 2018 at 12:34 PM zhen li <[hidden email]> wrote:

Hi all:
    How can I trigger the window manually in  fold operator or incremental aggregation? For example, when some conditions is meet，althouth the count window or time window is not reach

[hassahma]

Hi Niels,

Can we distinguish within apply function of 'RichWindowFunction' whether it was called due to onElement trigger call or onProcessingtime trigger call of a custom Trigger ?

Thanks!

On Wed, 17 Oct 2018 at 12:51, Niels van Kaam <[hidden email]> wrote:

Hi Zhen Li,

You can control when a windowed stream emits data with "Triggers". See: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#triggers 

Flink comes with a couple of default triggers, but you can also create your own by implementing https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/windowing/triggers/Trigger.html.

Note that this does not change the window, but just causes the windowedstream to emit intermediate results to the next operator.

Does this answer your question?

Cheers,

Niels

On Wed, Oct 17, 2018 at 12:34 PM zhen li <[hidden email]> wrote:

Hi all:
    How can I trigger the window manually in  fold operator or incremental aggregation? For example, when some conditions is meet，althouth the count window or time window is not reach

[Niels van Kaam]

Sorry, I would not know that. I have worked with custom triggers, but haven't actually had to implement a custom window function yet.

By looking at the interfaces I would not say that is possible.

Niels

On Wed, Oct 17, 2018 at 2:18 PM Ahmad Hassan <[hidden email]> wrote:

Hi Niels,

Can we distinguish within apply function of 'RichWindowFunction' whether it was called due to onElement trigger call or onProcessingtime trigger call of a custom Trigger ?

Thanks!

On Wed, 17 Oct 2018 at 12:51, Niels van Kaam <[hidden email]> wrote:

Hi Zhen Li,

You can control when a windowed stream emits data with "Triggers". See: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#triggers 

Flink comes with a couple of default triggers, but you can also create your own by implementing https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/windowing/triggers/Trigger.html.

Note that this does not change the window, but just causes the windowedstream to emit intermediate results to the next operator.

Does this answer your question?

Cheers,

Niels

On Wed, Oct 17, 2018 at 12:34 PM zhen li <[hidden email]> wrote:

Hi all:
    How can I trigger the window manually in  fold operator or incremental aggregation? For example, when some conditions is meet，althouth the count window or time window is not reach

[Dawid Wysakowicz-2]

Hi Zhen Li,

As far as I know that is not possible. For such custom handling I would recommend having a look at ProcessFunction[1], where you have access to timers and state.

Best,

Dawid

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/process_function.html#process-function-low-level-operations

On 17/10/2018 14:18, Ahmad Hassan wrote:

Hi Niels,

Can we distinguish within apply function of 'RichWindowFunction' whether it was called due to onElement trigger call or onProcessingtime trigger call of a custom Trigger ?

Thanks!

On Wed, 17 Oct 2018 at 12:51, Niels van Kaam <[hidden email]> wrote:

Hi Zhen Li,

You can control when a windowed stream emits data with "Triggers". See: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#triggers 

Flink comes with a couple of default triggers, but you can also create your own by implementing https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/windowing/triggers/Trigger.html.

Note that this does not change the window, but just causes the windowedstream to emit intermediate results to the next operator.

Does this answer your question?

Cheers,

Niels

On Wed, Oct 17, 2018 at 12:34 PM zhen li <[hidden email]> wrote:

Hi all:
    How can I trigger the window manually in  fold operator or incremental aggregation? For example, when some conditions is meet，althouth the count window or time window is not reach

[Dominik Wosiński]

Hey Zhen Li,

What are You trying to do exactly? Maybe there is a more suitable method than manually triggering windows available in Flink.

Best Regards,

Dom.

śr., 24 paź 2018 o 09:25 Dawid Wysakowicz <[hidden email]> napisał(a):

Hi Zhen Li,

As far as I know that is not possible. For such custom handling I would recommend having a look at ProcessFunction[1], where you have access to timers and state.

Best,

Dawid

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/process_function.html#process-function-low-level-operations

On 17/10/2018 14:18, Ahmad Hassan wrote:

Hi Niels,

Can we distinguish within apply function of 'RichWindowFunction' whether it was called due to onElement trigger call or onProcessingtime trigger call of a custom Trigger ?

Thanks!

On Wed, 17 Oct 2018 at 12:51, Niels van Kaam <[hidden email]> wrote:

Hi Zhen Li,

You can control when a windowed stream emits data with "Triggers". See: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#triggers 

Flink comes with a couple of default triggers, but you can also create your own by implementing https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/streaming/api/windowing/triggers/Trigger.html.

Note that this does not change the window, but just causes the windowedstream to emit intermediate results to the next operator.

Does this answer your question?

Cheers,

Niels

On Wed, Oct 17, 2018 at 12:34 PM zhen li <[hidden email]> wrote:

Hi all:
    How can I trigger the window manually in  fold operator or incremental aggregation? For example, when some conditions is meet，althouth the count window or time window is not reach