Usage of "onTime" in ProcessFunction

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Usage of "onTime" in ProcessFunction

Boris Lublinsky
I am effectively trying to simulate processing windows - drop the results that are not complete in time and was trying to use onTimer method in my Processor implementation.
I am not sure that I understand exactly how this works. When I start execution (in a different processor) I am executing
ctx.timerService.registerEventTimeTimer(currentTime + speculativeTimeout.value())
Basically the absolute cut off time.

Is this the right usage? What is happening when I have more then on timer started?


Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

Reply | Threaded
Open this post in threaded view
|

Re: Usage of "onTime" in ProcessFunction

Andrey Zagrebin
Hi,

the timers are scoped to the current key when you apply a processing function to a KeyedStream.
If you register more than one timer for a particular key and timestamp, you will get only one onTimer callback, see also in docs [1]. Timers registered in a processing function will trigger only in this processing function. All records and timer callbacks are processed sequentially for a particular key in one of parallel instances of the operator.

Depending on your use case, if you use event time timer, it might make sense to use current watermark as a ‘currentTime’ in your code snippet.

Best,
Andrey


On 31 Aug 2018, at 13:42, Boris Lublinsky <[hidden email]> wrote:

I am effectively trying to simulate processing windows - drop the results that are not complete in time and was trying to use onTimer method in my Processor implementation.
I am not sure that I understand exactly how this works. When I start execution (in a different processor) I am executing
ctx.timerService.registerEventTimeTimer(currentTime + speculativeTimeout.value())
Basically the absolute cut off time.

Is this the right usage? What is happening when I have more then on timer started?


Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/


Reply | Threaded
Open this post in threaded view
|

Re: Usage of "onTime" in ProcessFunction

Boris Lublinsky
Thanks Andrey
I do not have event time, dealing only with process time.
My process gets 2 types of messages:
1. Start processing, which starts the timer, creates a GUID and outputs event to another stream for the actual processing. Lets say at time 45s and I want to make sure that my result will come back in the next 10s, otherwise I ignore the response. I then start a timer for time 55s
2. Reply that can either come back in time or later. If it is in time (GUID is present), I send the reply back. If GUID is not present I Ignore it

onTimer is basically removing GUID that timed out from memory, so that I can ignore the late arrivals

Now I can have several start requests for times 45, 50, and 52. If my trout is ten, I have timers for 55, 60 and 62.

Will all of them fire at these time intervals assuming that timer’s processing time is 0?




Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/

On Aug 31, 2018, at 2:20 PM, Andrey Zagrebin <[hidden email]> wrote:

Hi,

the timers are scoped to the current key when you apply a processing function to a KeyedStream.
If you register more than one timer for a particular key and timestamp, you will get only one onTimer callback, see also in docs [1]. Timers registered in a processing function will trigger only in this processing function. All records and timer callbacks are processed sequentially for a particular key in one of parallel instances of the operator.

Depending on your use case, if you use event time timer, it might make sense to use current watermark as a ‘currentTime’ in your code snippet.

Best,
Andrey


On 31 Aug 2018, at 13:42, Boris Lublinsky <[hidden email]> wrote:

I am effectively trying to simulate processing windows - drop the results that are not complete in time and was trying to use onTimer method in my Processor implementation.
I am not sure that I understand exactly how this works. When I start execution (in a different processor) I am executing
ctx.timerService.registerEventTimeTimer(currentTime + speculativeTimeout.value())
Basically the absolute cut off time.

Is this the right usage? What is happening when I have more then on timer started?


Boris Lublinsky
FDP Architect
[hidden email]
https://www.lightbend.com/