Usecase for Flink

[igor.berman]

Hi,
We are looking at Flink and trying to understand if our usecase is relevant to it.

We need process stream of events. Each event is for some id(e.g. device id), when each event should be
1. stored in some persistent storage(e.g. cassandra)
2. previously persisted events should be fetched and some computation over whole history may or may not trigger some other events(e.g. sending email)

so yes we have stream of events, but we need persistent store(aka external service) in the middle
and there is no aggregation of those events into something smaller which could be stored in memory, i.e. number of ids might be huge and previous history of events per each id can be considerable so that no way to store everything in memory

I was wondering if akka stream is sort of optional solution too

please share your ideas :)
thanks in advance,
Igor

[Stephan Ewen]

If I understand you correctly, you want to write something like:

------------------------------------------------------------------

                                                    [cassandra]

                              ^

                              |

                              V

(event source) ----> (Add event and lookup) ---> (further ops)

------------------------------------------------------------------

That should work with Flink, yes. You can communicate with an external Cassandra service inside functions.

We are also working on making larger-than-memory state easily supported in Flink, so future versions may allow you to

do this without any external service.

On Thu, Dec 17, 2015 at 8:54 AM, igor.berman <[hidden email]> wrote:

Hi,
We are looking at Flink and trying to understand if our usecase is relevant
to it.

We need process stream of events. Each event is for some id(e.g. device id),
when each event should be
1. stored in some persistent storage(e.g. cassandra)
2. previously persisted events should be fetched and some computation over
whole history may or may not trigger some other events(e.g. sending email)

so yes we have stream of events, but we need persistent store(aka external
service) in the middle
and there is no aggregation of those events into something smaller which
could be stored in memory, i.e. number of ids might be huge and previous
history of events per each id can be considerable so that no way to store
everything in memory

I was wondering if akka stream is sort of optional solution too

please share your ideas :)
thanks in advance,
Igor



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usecase-for-Flink-tp4076.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[igor.berman]

thanks Stephan,
yes, you got usecase right