How to clear state immediately after a keyed window is processed?

[HarshithBolar]

Hi all,

 

My application uses a keyed window that is keyed by a function of timestamp. This means once that particular window has been fired and processed, there is no use in keeping that key active because there is no way that particular key will appear again. Because this use case involves continuously expanding keys, I want to clear the state of a key as soon as it is finished processing without having to configure timers.

 

Is this something that can be achieved in the evictor method or apply method after each keyed window is done processing?

 

Thanks,

Harshith

 

 

[Konstantin Knauf-2]

Hi Harshith,

when you use Flink's Windowing API, the state of an event time window is cleared once the watermark passes the end time of the window (that's when the window fires) + the allowed lateness. So, as long as you don't configure additional allowed lateness (default=0), Flink will already behave as described by default.

Cheers,

Konstantin

On Thu, Feb 14, 2019 at 12:03 PM Kumar Bolar, Harshith <[hidden email]> wrote:

Hi all,

 

My application uses a keyed window that is keyed by a function of timestamp. This means once that particular window has been fired and processed, there is no use in keeping that key active because there is no way that particular key will appear again. Because this use case involves continuously expanding keys, I want to clear the state of a key as soon as it is finished processing without having to configure timers.

 

Is this something that can be achieved in the evictor method or apply method after each keyed window is done processing?

 

Thanks,

Harshith

 

 

--

[Konstantin Knauf-2]

Yes, for processing-time windows the clean up time is exactly the end time of the window, because by definition there is no late data and state does not need to be kept around.

On Thu, Feb 14, 2019 at 1:03 PM Kumar Bolar, Harshith <[hidden email]> wrote:

Thanks Konstanin,

 

But I’m using processing time, hence no watermarks. Will the state still be cleared automatically if nothing is done?

 

From: Konstantin Knauf <[hidden email]>
Date: Thursday, 14 February 2019 at 5:18 PM
To: Harshith Kumar Bolar <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: [External] Re: How to clear state immediately after a keyed window is processed?

 

Hi Harshith,

 

when you use Flink's Windowing API, the state of an event time window is cleared once the watermark passes the end time of the window (that's when the window fires) + the allowed lateness. So, as long as you don't configure additional allowed lateness (default=0), Flink will already behave as described by default.

 

Cheers,

 

Konstantin

 

On Thu, Feb 14, 2019 at 12:03 PM Kumar Bolar, Harshith <[hidden email]> wrote:

Hi all,

 

My application uses a keyed window that is keyed by a function of timestamp. This means once that particular window has been fired and processed, there is no use in keeping that key active because there is no way that particular key will appear again. Because this use case involves continuously expanding keys, I want to clear the state of a key as soon as it is finished processing without having to configure timers.

 

Is this something that can be achieved in the evictor method or apply method after each keyed window is done processing?

 

Thanks,

Harshith

 

 

--

Konstantin Knauf | Solutions Architect

[phone]

 

 

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen   

--

[HarshithBolar]

Thanks a lot :D

 

From: Konstantin Knauf <[hidden email]>
Date: Thursday, 14 February 2019 at 5:38 PM
To: Harshith Kumar Bolar <[hidden email]>, user <[hidden email]>
Subject: [External] Re: Re: How to clear state immediately after a keyed window is processed?

 

Yes, for processing-time windows the clean up time is exactly the end time of the window, because by definition there is no late data and state does not need to be kept around.

 

On Thu, Feb 14, 2019 at 1:03 PM Kumar Bolar, Harshith <[hidden email]> wrote:

Thanks Konstanin,

 

But I’m using processing time, hence no watermarks. Will the state still be cleared automatically if nothing is done?

 

From: Konstantin Knauf <[hidden email]>
Date: Thursday, 14 February 2019 at 5:18 PM
To: Harshith Kumar Bolar <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: [External] Re: How to clear state immediately after a keyed window is processed?

 

Hi Harshith,

 

when you use Flink's Windowing API, the state of an event time window is cleared once the watermark passes the end time of the window (that's when the window fires) + the allowed lateness. So, as long as you don't configure additional allowed lateness (default=0), Flink will already behave as described by default.

 

Cheers,

 

Konstantin

 

On Thu, Feb 14, 2019 at 12:03 PM Kumar Bolar, Harshith <[hidden email]> wrote:

Hi all,

 

My application uses a keyed window that is keyed by a function of timestamp. This means once that particular window has been fired and processed, there is no use in keeping that key active because there is no way that particular key will appear again. Because this use case involves continuously expanding keys, I want to clear the state of a key as soon as it is finished processing without having to configure timers.

 

Is this something that can be achieved in the evictor method or apply method after each keyed window is done processing?

 

Thanks,

Harshith

 

 

--

Konstantin Knauf | Solutions Architect

[phone]

 

Error! Filename not specified.

 

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen   

--

Konstantin Knauf | Solutions Architect

[phone]

 

 

Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen