Custom timer implementation using Flink

[jaxbihani]

I have a use case which I am trying to solve using Flink. Need an advice to decide on the correct approach.

Use case:
----------
 I have a stream of events partitioned by a key. For some events, I need to start a timer (conside this as a SLA i.e. if something is not done in x secs/minutes do something). Now when that timer expires I need to perform some arbitrary action (like writing to database etc). There will be some events which can cancel the timers. (i.e. if event comes before x secs we need not run SLA violation action etc.). We are considering flink because then we can reuse the scaling, fault tolerance provided by the engine rather than building our own. Current rps is ~ 200-300 but it can be expected to increase quickly.

Solutions in mind:
-------------------
1. We can think it like CEP use case, where with encoding like "event1 followed by event2" with "not" in x seconds. i.e. when event 2 is "not" arrived in x seconds. I assume there will be NOT operator support. I am not sure about memory consumption in CEP. Because x seconds can be x days as well and I do not need any batching of events in memory. I just need to start a timer of x days/hours (lets say) and when it is fired just trigger something. So there is no notion of window as such. Can CEP fit in this type of use case? If the  timer between events is in days, how about the memory consumption?

2. Use Flink for event processing and delegate the tasks of timers to another service i.e. when event occurs send it to kafka with timer information and then another service handles timers and send back the event again once that is done etc.  Looks like many hops in this process and latency will be high if SLA is in seconds (I am thinking of using Kafka here).

Is anyone aware of a better way of doing this in flink?

[Tzu-Li (Gordon) Tai]

Hi,

I just need to 
start a timer of x days/hours (lets say) and when it is fired just trigger 
something.

Flink’s lower-level ProcessFunction [1] should be very suitable to implement this. Have you taken a look at this and see if it suits your case?

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/process_function.html

Cheers,

Gordon

On April 11, 2017 at 3:25:39 AM, jaxbihani ([hidden email]) wrote:

I have a use case which I am trying to solve using Flink. Need an advice to
decide on the correct approach.

Use case:
----------
I have a stream of events partitioned by a key. For some events, I need to
start a timer (conside this as a SLA i.e. if something is not done in x
secs/minutes do something). Now when that timer expires I need to perform
some arbitrary action (like writing to database etc). There will be some
events which can cancel the timers. (i.e. if event comes before x secs we
need not run SLA violation action etc.). We are considering flink because
then we can reuse the scaling, fault tolerance provided by the engine rather
than building our own. Current rps is ~ 200-300 but it can be expected to
increase quickly.

Solutions in mind:
-------------------
1. We can think it like CEP use case, where with encoding like "event1
followed by event2" with "not" in x seconds. i.e. when event 2 is "not"
arrived in x seconds. I assume there will be NOT operator support. I am not
sure about memory consumption in CEP. Because x seconds can be x days as
well and I do not need any batching of events in memory. I just need to
start a timer of x days/hours (lets say) and when it is fired just trigger
something. So there is no notion of window as such. Can CEP fit in this type
of use case? If the timer between events is in days, how about the memory
consumption?

2. Use Flink for event processing and delegate the tasks of timers to
another service i.e. when event occurs send it to kafka with timer
information and then another service handles timers and send back the event
again once that is done etc. Looks like many hops in this process and
latency will be high if SLA is in seconds (I am thinking of using Kafka
here).

Is anyone aware of a better way of doing this in flink?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-timer-implementation-using-Flink-tp12581.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[jaxbihani]

Hi Gordon

I somehow missed this in the docs. Looks really helpful for this use case. Thanks.
I now have some other ideas for this use case. I will post here if I use this or not. If not, will post the approach which I will take.