Initializing global data

[nsengupta]

Hello Flinksters,

 

I need to initialize a piece of global data at the beginning of a Flink application. I use this piece as a READ-ONLY for enhancing the streamed data that come in at the time of execution. So, this piece of data also has to be present in all the execution nodes. For readability, let's name this piece as ReferableData.

The ReferableData is essentially a Map[String, UserDefinedClass]. The source is a regular CSV file, present in the local file system (no HDFS,for the time being). So, I read the CSV file in the usual manner, parse it, create instances of UserDefinedClass and create a Map[asStringK, asInstanceV] and make this available globally. 

What I gather from the documents and discussions in this forum is that one of the ways to achieve this is to use

env.getConfig().setGlobalJobParameters(parameters)

Where 'parameters' is the Map is the ReferableData that I want to create from the CSV file.

I have two questions:

a) Is this approach of providing look-up data to all nodes, a good practice?

b) Should I read the input CSV file through the ExecutionEnvironment.readTextFile or using standard IO routines of Java? Does the community have any preference?

-- Nirmalya

--

Nirmalya Sengupta about.me/sengupta.nirmalya

[stefanobaghino]

Hi Nirmalya,

I'm not really sure setGlobalJobParameters is what you're looking for. If the ReferableData more then some simple configuration (and judging from its type it looks like so) maybe you can try to leverage broadcast variables. You can read more about them here: https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/index.html#broadcast-variables

Regarding your question on CSV, I always find the built-in APIs very handy. I also expect them to work in a parallel fashion on distributed file systems out of the box, so I wouldn't re-write them (unless you have very specific needs, of course).

On Tue, Apr 26, 2016 at 3:25 PM, Nirmalya Sengupta <[hidden email]> wrote:

Hello Flinksters,

 

I need to initialize a piece of global data at the beginning of a Flink application. I use this piece as a READ-ONLY for enhancing the streamed data that come in at the time of execution. So, this piece of data also has to be present in all the execution nodes. For readability, let's name this piece as ReferableData.

The ReferableData is essentially a Map[String, UserDefinedClass]. The source is a regular CSV file, present in the local file system (no HDFS,for the time being). So, I read the CSV file in the usual manner, parse it, create instances of UserDefinedClass and create a Map[asStringK, asInstanceV] and make this available globally. 

What I gather from the documents and discussions in this forum is that one of the ways to achieve this is to use

env.getConfig().setGlobalJobParameters(parameters)

Where 'parameters' is the Map is the ReferableData that I want to create from the CSV file.

I have two questions:

a) Is this approach of providing look-up data to all nodes, a good practice?

b) Should I read the input CSV file through the ExecutionEnvironment.readTextFile or using standard IO routines of Java? Does the community have any preference?

-- Nirmalya

--

Software Technologist
http://www.linkedin.com/in/nirmalyasengupta
"If you have built castles in the air, your work need not be lost. That is where they should be.
Now put the foundation under them."

Nirmalya Sengupta about.me/sengupta.nirmalya

--

[nsengupta]

Hello Stefano <[hidden email]>,

Thanks for sharing your views. Now, that you make me think, I know that your recommendation works well. 

I will go ahead, following your suggestions.

-- Nirmalya

--

Nirmalya Sengupta about.me/sengupta.nirmalya