Passing Configuration & State

[Navneeth Krishnan]

Hi All,

I have developed a streaming pipeline in java and I need to pass some of the configuration parameters that are passed during program startup to user functions. I used the below link as reference.

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_practices.html

I have tried withParameters & setGlobalJobParameters but that doesn't seem to work. The parameters are blank inside my user function when deployed in a cluster. I have also tried passing the parameters inside the constructor of my user function and this seem to work on local but not in cluster mode, again the parameters are blank.

Is there a recommended way to pass the program parameters to user function classes?

Also, I have scenario where the state created inside a user function has to passed around to multiple classes. Is there a state registry or something from which I can retrieve a registered state and use or should I implement my own?

Thanks in advance.

Regards,

Navneeth

[Fabian Hueske-2]

Hi Navneeth,

the configuring user function using a Configuration object and setting the parameters in the open() method of a RichFunction is no longer recommended.

In fact, that only works for the DataSet API and has not been added for the DataStream API. The open() method with the Configuration parameter still exists because DataSet and DataStream API share these interfaces.

The recommended way to configure user functions is via constructor parameters while defining the streaming application. The function object that you create there is serialize and shipped to the workers.

Hence, you can do something like:

DataStream<X> input = ...

DataStream<Y> output = input.map(new MyMapFunction(param1, param2));

where param1 and param2 are stored in MyMapFunction, serialized and shipped with the MyMapFunction instance.

Regarding sharing state objects inside of a user function. You can access the state object from the RuntimeContext.
However, you must make sure that the state is only accessed within function calls of the user function, i.e, you should not leak it to other operator through a singleton that holds state objects.

Hope this helps,

Fabian

2017-10-26 16:54 GMT+02:00 Navneeth Krishnan <[hidden email]>:

Hi All,

I have developed a streaming pipeline in java and I need to pass some of the configuration parameters that are passed during program startup to user functions. I used the below link as reference.

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_practices.html

I have tried withParameters & setGlobalJobParameters but that doesn't seem to work. The parameters are blank inside my user function when deployed in a cluster. I have also tried passing the parameters inside the constructor of my user function and this seem to work on local but not in cluster mode, again the parameters are blank.

Is there a recommended way to pass the program parameters to user function classes?

Also, I have scenario where the state created inside a user function has to passed around to multiple classes. Is there a state registry or something from which I can retrieve a registered state and use or should I implement my own?

Thanks in advance.

Regards,

Navneeth