Control triggering on empty window

[Maxim]

I have the following use case:

Input stream of timestamped "on" and "off" events received out of order.

I need to produce an event with time that system was "on" every 15 minutes. Events should be produced only for intervals that system was "on".

When 15 minute window has at least one record it is triggered and the required aggregate is created, but when no event is received within 15 minute period window is not triggered and nothing is produced.

I understand that it is not feasible to trigger on empty windows when the set of keys is unbounded. But it would be nice to give the control for such triggering to a window function. In my case the window function could enable the empty triggering for the current key when the last event in the evaluated window is "on" and disable it if is "off". 

The strawman API for such feature:

public void apply(String key, TimeWindow window, Iterable<OnOffEvent> input, Collector<Aggregate> out) throws Exception {

    ...    

    RuntimeContext context = this.getRuntimeContext();

    if (lastEvent.isOn()) {

       context.enableEmptyWindowTriggering();

    } else {

       context.disableEmptyWindowTriggering();

    }

}

I could implement the same logic using global window and custom trigger and evictor, but it looks like ugly workaround to me.

Is there any better way to solve this use case?

Thanks,

Maxim.

[Aljoscha Krettek]

Hi,

I'm afraid this is not possible with our windowing model (expect with hacks using GlobalWindow, as you mentioned). The reason is, that windows only come into existence once there is an element that has a window. Before that, the system has no reference point about what windows there should exist because there is no knowledge about time except when looking at elements.

Cheers,

Aljoscha

On Thu, 21 Apr 2016 at 01:31 Maxim <[hidden email]> wrote:

I have the following use case:

Input stream of timestamped "on" and "off" events received out of order.

I need to produce an event with time that system was "on" every 15 minutes. Events should be produced only for intervals that system was "on".

When 15 minute window has at least one record it is triggered and the required aggregate is created, but when no event is received within 15 minute period window is not triggered and nothing is produced.

I understand that it is not feasible to trigger on empty windows when the set of keys is unbounded. But it would be nice to give the control for such triggering to a window function. In my case the window function could enable the empty triggering for the current key when the last event in the evaluated window is "on" and disable it if is "off". 

The strawman API for such feature:

public void apply(String key, TimeWindow window, Iterable<OnOffEvent> input, Collector<Aggregate> out) throws Exception {

    ...    

    RuntimeContext context = this.getRuntimeContext();

    if (lastEvent.isOn()) {

       context.enableEmptyWindowTriggering();

    } else {

       context.disableEmptyWindowTriggering();

    }

}

I could implement the same logic using global window and custom trigger and evictor, but it looks like ugly workaround to me.

Is there any better way to solve this use case?

Thanks,

Maxim.

[Maxim]

I think the best way to support such a feature is to extend WindowAssigner with ability to be called on timer and checkpoint its state the same way it is done by the Trigger.

Such WindowAssigner would be able to create Windows based on time even if no event is received.

On Thu, Apr 21, 2016 at 1:57 AM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

I'm afraid this is not possible with our windowing model (expect with hacks using GlobalWindow, as you mentioned). The reason is, that windows only come into existence once there is an element that has a window. Before that, the system has no reference point about what windows there should exist because there is no knowledge about time except when looking at elements.

Cheers,

Aljoscha

On Thu, 21 Apr 2016 at 01:31 Maxim <[hidden email]> wrote:

I have the following use case:

Input stream of timestamped "on" and "off" events received out of order.

I need to produce an event with time that system was "on" every 15 minutes. Events should be produced only for intervals that system was "on".

When 15 minute window has at least one record it is triggered and the required aggregate is created, but when no event is received within 15 minute period window is not triggered and nothing is produced.

I understand that it is not feasible to trigger on empty windows when the set of keys is unbounded. But it would be nice to give the control for such triggering to a window function. In my case the window function could enable the empty triggering for the current key when the last event in the evaluated window is "on" and disable it if is "off". 

The strawman API for such feature:

public void apply(String key, TimeWindow window, Iterable<OnOffEvent> input, Collector<Aggregate> out) throws Exception {

    ...    

    RuntimeContext context = this.getRuntimeContext();

    if (lastEvent.isOn()) {

       context.enableEmptyWindowTriggering();

    } else {

       context.disableEmptyWindowTriggering();

    }

}

I could implement the same logic using global window and custom trigger and evictor, but it looks like ugly workaround to me.

Is there any better way to solve this use case?

Thanks,

Maxim.

[Aljoscha Krettek]

Hi,

I'm sorry but I still think this is not possible. Windows are usually associated with a key, so if there is no element to which we can assign a window then there is also no key to which the window would belong.

Cheers,

Aljoscha

On Thu, 21 Apr 2016 at 22:35 Maxim <[hidden email]> wrote:

I think the best way to support such a feature is to extend WindowAssigner with ability to be called on timer and checkpoint its state the same way it is done by the Trigger.

Such WindowAssigner would be able to create Windows based on time even if no event is received.

On Thu, Apr 21, 2016 at 1:57 AM, Aljoscha Krettek <[hidden email]> wrote:

Hi,

I'm afraid this is not possible with our windowing model (expect with hacks using GlobalWindow, as you mentioned). The reason is, that windows only come into existence once there is an element that has a window. Before that, the system has no reference point about what windows there should exist because there is no knowledge about time except when looking at elements.

Cheers,

Aljoscha

On Thu, 21 Apr 2016 at 01:31 Maxim <[hidden email]> wrote:

I have the following use case:

Input stream of timestamped "on" and "off" events received out of order.

I need to produce an event with time that system was "on" every 15 minutes. Events should be produced only for intervals that system was "on".

When 15 minute window has at least one record it is triggered and the required aggregate is created, but when no event is received within 15 minute period window is not triggered and nothing is produced.

I understand that it is not feasible to trigger on empty windows when the set of keys is unbounded. But it would be nice to give the control for such triggering to a window function. In my case the window function could enable the empty triggering for the current key when the last event in the evaluated window is "on" and disable it if is "off". 

The strawman API for such feature:

public void apply(String key, TimeWindow window, Iterable<OnOffEvent> input, Collector<Aggregate> out) throws Exception {

    ...    

    RuntimeContext context = this.getRuntimeContext();

    if (lastEvent.isOn()) {

       context.enableEmptyWindowTriggering();

    } else {

       context.disableEmptyWindowTriggering();

    }

}

I could implement the same logic using global window and custom trigger and evictor, but it looks like ugly workaround to me.

Is there any better way to solve this use case?

Thanks,

Maxim.