Random access to small global state

[Sebastian Schelter-2]

Hi,

I'm planning to work on a streaming recommender in Flink, and one
problem that I have is that the algorithm needs random access to a small
global state (say a million counts). It should be ok if there is some
inconsistency in the state (e.g., delay in seeing updates).

Does anyone here have experience with such things? I'm thinking of
connecting Flink to a lighweight in-memory key-value store such as
memcache for that.

Best,
Sebastian

[Suneel Marthi-2]

U could use ignite too, I believe they have a plugin for flink streaming.

Sent from my iPhone

> On Jul 9, 2016, at 8:05 AM, Sebastian <[hidden email]> wrote:
>
> Hi,
>
> I'm planning to work on a streaming recommender in Flink, and one problem that I have is that the algorithm needs random access to a small global state (say a million counts). It should be ok if there is some inconsistency in the state (e.g., delay in seeing updates).
>
> Does anyone here have experience with such things? I'm thinking of connecting Flink to a lighweight in-memory key-value store such as memcache for that.
>
> Best,
> Sebastian

[rmetzger0]

Hi,

For Ignite, Flink has a Sink, which is a one-directional thing.

I think that Sebastian needs a bi-directional connection.

An in-memory KV store like redis or memcache is probably the best option for such a use case (it reminds me a bit of the Yahoo streaming benchmark [1]).

[1] https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at

On Sat, Jul 9, 2016 at 4:03 PM, Suneel Marthi <[hidden email]> wrote:

U could use ignite too, I believe they have a plugin for flink streaming.

Sent from my iPhone

> On Jul 9, 2016, at 8:05 AM, Sebastian <[hidden email]> wrote:
>
> Hi,
>
> I'm planning to work on a streaming recommender in Flink, and one problem that I have is that the algorithm needs random access to a small global state (say a million counts). It should be ok if there is some inconsistency in the state (e.g., delay in seeing updates).
>
> Does anyone here have experience with such things? I'm thinking of connecting Flink to a lighweight in-memory key-value store such as memcache for that.
>
> Best,
> Sebastian