externalizing config flies for flink class loader

[Vishwas Siravara]

I have a standalone cluster. I have added my own library(jar file) to the lib/ folder in flink . I submit my job from cli after I start the cluster. Now I want to externalize a property file which has to be read by this library. Since this library is loaded by flink's classloader and not the application class loader I cannot supply this using flink run -C ... since this works only for user class loader. 

Thanks,

Vishwas 

[Vijay Bhaskar]

Hi

You can use this way:

Use typesafe configuration, which provides excellent configuration methodologies. 

You supply default configuration, which is read by your application through reference.conf file of typesafe. If you want to override any of the defaults you can supply to command line arguments

From the command line arguments, you can read them into Paramter tool map, which is supplied by flink. You can use this map to override the default configuration of the typesafe. 

Following is the code snippet:

val pmtool = ParameterTool.fromArgs(args)

val defaultConfig = ConfigFactory.load() //Default config in reference.conf/application.conf/system properties/env  of typesafe

val overrideConfigFromArgs =  ConfigFactory.load(pmtool.toMap)

val finalConfig =  overrideConfigFromArgs.withFallBack(defaultConfig) //This is going to override your command line params

Regards

Bhaskar

On Fri, Sep 13, 2019 at 12:51 AM Vishwas Siravara <[hidden email]> wrote:

I have a standalone cluster. I have added my own library(jar file) to the lib/ folder in flink . I submit my job from cli after I start the cluster. Now I want to externalize a property file which has to be read by this library. Since this library is loaded by flink's classloader and not the application class loader I cannot supply this using flink run -C ... since this works only for user class loader. 

Thanks,

Vishwas 

[Vijay Bhaskar]

Sorry there is a typo, corrected it:

val pmtool = ParameterTool.fromArgs(args)

val defaultConfig = ConfigFactory.load() //Default config in reference.conf/application.conf/system properties/env  of typesafe

val overrideConfigFromArgs =  ConfigFactory.load(pmtool.toMap)

val finalConfig =  overrideConfigFromArgs.withFallBack(defaultConfig) //This is going to override your default configuration  params

On Fri, Sep 13, 2019 at 11:38 AM Vijay Bhaskar <[hidden email]> wrote:

Hi

You can use this way:

Use typesafe configuration, which provides excellent configuration methodologies. 

You supply default configuration, which is read by your application through reference.conf file of typesafe. If you want to override any of the defaults you can supply to command line arguments

From the command line arguments, you can read them into Paramter tool map, which is supplied by flink. You can use this map to override the default configuration of the typesafe. 

Following is the code snippet:

val pmtool = ParameterTool.fromArgs(args)

val defaultConfig = ConfigFactory.load() //Default config in reference.conf/application.conf/system properties/env  of typesafe

val overrideConfigFromArgs =  ConfigFactory.load(pmtool.toMap)

val finalConfig =  overrideConfigFromArgs.withFallBack(defaultConfig) //This is going to override your command line params

Regards

Bhaskar

On Fri, Sep 13, 2019 at 12:51 AM Vishwas Siravara <[hidden email]> wrote:

I have a standalone cluster. I have added my own library(jar file) to the lib/ folder in flink . I submit my job from cli after I start the cluster. Now I want to externalize a property file which has to be read by this library. Since this library is loaded by flink's classloader and not the application class loader I cannot supply this using flink run -C ... since this works only for user class loader. 

Thanks,

Vishwas