Event time window eviction

[Navneeth Krishnan]

Hi All,

I'm working on a very short tumbling window for 1 second per key. What I want to achieve is if the event time per key doesn't progress after a second I want to evict the window, basically a combination of event time and processing time. I'm currently achieving it by registering a processing time timer but is there a way to emit some global punctuator which can be used to evict all keys window data. 

The issue with registering processing time timer for every key is causing too much JVM pressure. Any suggestions on how this could be implemented?

Thanks

[Navneeth Krishnan]

Hi All,

Any suggestions?

Thanks

On Thu, Jul 25, 2019 at 11:45 PM Navneeth Krishnan <[hidden email]> wrote:

Hi All,

I'm working on a very short tumbling window for 1 second per key. What I want to achieve is if the event time per key doesn't progress after a second I want to evict the window, basically a combination of event time and processing time. I'm currently achieving it by registering a processing time timer but is there a way to emit some global punctuator which can be used to evict all keys window data. 

The issue with registering processing time timer for every key is causing too much JVM pressure. Any suggestions on how this could be implemented?

Thanks

[taher koitawala-2]

I believe the approach to this is wrong... For fixing windows we can write our custom triggers to fire them... However what I'm not convinced with is switching between event and processing time. 

         Write a custom triggers and fire the event time window if you don't see any activity. That's the only way. 

On Mon, Jul 29, 2019, 11:07 PM Navneeth Krishnan <[hidden email]> wrote:

Hi All,

Any suggestions?

Thanks

On Thu, Jul 25, 2019 at 11:45 PM Navneeth Krishnan <[hidden email]> wrote:

Hi All,

I'm working on a very short tumbling window for 1 second per key. What I want to achieve is if the event time per key doesn't progress after a second I want to evict the window, basically a combination of event time and processing time. I'm currently achieving it by registering a processing time timer but is there a way to emit some global punctuator which can be used to evict all keys window data. 

The issue with registering processing time timer for every key is causing too much JVM pressure. Any suggestions on how this could be implemented?

Thanks

[Navneeth Krishnan]

Thanks Taher. Are there any examples for this? In my scenario I would have data coming in and it might stop for sometime but I need the window to end after the duration.

Also, I believe in version 1.3 the event time will progress only if all partitions in a kafka topic pass the event time. Is that still the case? If there is data in only few partitions will the event time progress?

Thanks

On Mon, Jul 29, 2019 at 10:51 AM taher koitawala <[hidden email]> wrote:

I believe the approach to this is wrong... For fixing windows we can write our custom triggers to fire them... However what I'm not convinced with is switching between event and processing time. 

         Write a custom triggers and fire the event time window if you don't see any activity. That's the only way. 

On Mon, Jul 29, 2019, 11:07 PM Navneeth Krishnan <[hidden email]> wrote:

Hi All,

Any suggestions?

Thanks

On Thu, Jul 25, 2019 at 11:45 PM Navneeth Krishnan <[hidden email]> wrote:

Hi All,

I'm working on a very short tumbling window for 1 second per key. What I want to achieve is if the event time per key doesn't progress after a second I want to evict the window, basically a combination of event time and processing time. I'm currently achieving it by registering a processing time timer but is there a way to emit some global punctuator which can be used to evict all keys window data. 

The issue with registering processing time timer for every key is causing too much JVM pressure. Any suggestions on how this could be implemented?

Thanks

[taher koitawala-2]

Hi Navneeth,

      There are 3 ways we can work with data now flowing and windows and not being fired because watermark flow stopped.

1. Write a custom trigger which fires the window if no elements arrive.

2. Your watermark assigner function can also house the logic that if no more watermarks are flowing, explicitly generate a higher watermark and force windows to fire.

3. The new way since Flink 1.3 I suppose is to use SourceContext functions markAsTemporarilyIdle function. Doc - https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/streaming/api/functions/source/SourceFunction.SourceContext.html

As per examples, I am not really sure if there is one documented for these scenarios.

On Tue, Jul 30, 2019, 9:24 AM Navneeth Krishnan <[hidden email]> wrote:

Thanks Taher. Are there any examples for this? In my scenario I would have data coming in and it might stop for sometime but I need the window to end after the duration.

Also, I believe in version 1.3 the event time will progress only if all partitions in a kafka topic pass the event time. Is that still the case? If there is data in only few partitions will the event time progress?

Thanks

On Mon, Jul 29, 2019 at 10:51 AM taher koitawala <[hidden email]> wrote:

I believe the approach to this is wrong... For fixing windows we can write our custom triggers to fire them... However what I'm not convinced with is switching between event and processing time. 

         Write a custom triggers and fire the event time window if you don't see any activity. That's the only way. 

On Mon, Jul 29, 2019, 11:07 PM Navneeth Krishnan <[hidden email]> wrote:

Hi All,

Any suggestions?

Thanks

On Thu, Jul 25, 2019 at 11:45 PM Navneeth Krishnan <[hidden email]> wrote:

Hi All,

I'm working on a very short tumbling window for 1 second per key. What I want to achieve is if the event time per key doesn't progress after a second I want to evict the window, basically a combination of event time and processing time. I'm currently achieving it by registering a processing time timer but is there a way to emit some global punctuator which can be used to evict all keys window data. 

The issue with registering processing time timer for every key is causing too much JVM pressure. Any suggestions on how this could be implemented?

Thanks