Looking up values in a metadata store for "condensed" events

[milind parikh]

I am new to flink. I am wondering what the best practice is for looking up additional values for entities embedded in an event.

For example, an event might have the engine model #. A metadata store has the typical rpm characteristics of an engine model #; which can be used in a subsequent processing of the event. I call the process of event inflation  "event adornment". Currently I use a Map function transformation to do the lookup in the Map function through instatiating a connection in the Map function to the Metadata store per map execution. The thing works; however having 100K connections is not very ideal (assuming 100k eps). Any alternative ideas; given the implementing class has to be Serializable?

[Aljoscha Krettek]

Hi,

what you can also do is use a RichMapFunction. Rich functions have open()/close() methods that get called before the first element and the last element, respectively.

See here https://ci.apache.org/projects/flink/flink-docs-master/apis/common/index.html#specifying-transformation-functions for more information.

Cheers,

Aljoscha

On Tue, 10 May 2016 at 07:31 milind parikh <[hidden email]> wrote:

I am new to flink. I am wondering what the best practice is for looking up additional values for entities embedded in an event.

For example, an event might have the engine model #. A metadata store has the typical rpm characteristics of an engine model #; which can be used in a subsequent processing of the event. I call the process of event inflation  "event adornment". Currently I use a Map function transformation to do the lookup in the Map function through instatiating a connection in the Map function to the Metadata store per map execution. The thing works; however having 100K connections is not very ideal (assuming 100k eps). Any alternative ideas; given the implementing class has to be Serializable?