Re: feature request: broadcast POJO objects as part of runtime context

Posted by Flavio Pompermaier on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/feature-request-broadcast-POJO-objects-as-part-of-runtime-context-tp367p372.html

I think Stefano was asking for a different way to pass a generic Configuration object, not just a subclass of it.
For example, in our use case, it would be helpful to broadcast around generic objects/pojos.
Is that possible?

On Nov 10, 2014 6:46 PM, "Stephan Ewen" <[hidden email]> wrote:
Hi!

I hope I understand correctly what you are trying to do: To have a config file available in the functions, you can simply do wither of the following:

-----------------------
Closure
-----------------------

Configuration conf = ...

data.map(new RichMapFunction<String, Integer>() {

  public void open (Conficuration c) {
     // access the conf object here
     conf.getString(...);
  }

  public Integer map(String value) {
    // whatever
  }
});
-----------------------


-----------------------
Config Parameters
-----------------------

Configuration conf = ...

data.map(new RichMapFunction<String, Integer>() {

  public void open (Conficuration c) {
     // access the c - it will will have all elements of the conf - see withParameters() below
     c.getString(...);
  }

  public Integer map(String value) {
    // whatever
  }
})
.withParameters(conf);
-----------------------


Stephan


On Mon, Nov 10, 2014 at 5:36 PM, Stefano Bortoli <[hidden email]> wrote:
Hi,

trying to run some legacy code as part of Flink Job, I had to replicate configurations files across my cluster. Not a big deal with a small cluster, but it would be nice to have these configuration objects broadcast-able. Namely, it would be nice to reuse the old "read from conf file" logic to build objects that then could be serialized and used along the processing through the broadcast mechanism.

Do you think it will be possible? with the new Kryo serialization it should not be extremely complicated.

saluti,
Stefano