Import Configuration File in Flink Cluster
Locked
Import Configuration File in Flink Cluster
 
|
Hi together
Currently I'm using flink on a docker cluster on AWS. I would like to use property files with the integrated ParameterTool.fromPropertiesFile function of Flink. Locally this version works absolutely fine: val configuration = ParameterTool.fromPropertiesFile("src/main/resources/config.properties") But on the cluster this didn't work, so we introduced this version, which also doesn't work: val baseParameters = ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile) gives java.io.FileNotFoundException: Properties file file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties does not exist The property file is located in src/main/resources. Do you have any idea of how to integrate such property files into the jar package? -Thanks Simon
|
Re: Import Configuration File in Flink Cluster
|
|
Hi Simon,
You'll have to write the property file to disk first to load it using the ParameterTool.fromPropertiesFile method. For example: // copy config from Java resource to a file File configOnDisk = new File("/path/to/config.properties"); Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"), configOnDisk.toPath()); // load the new file ParameterTool.fromPropertiesFile(configOnDisk); Cheers, Max On Mon, May 23, 2016 at 3:56 PM, simon peyer <[hidden email]> wrote: > Hi together > > Currently I'm using flink on a docker cluster on AWS. > I would like to use property files with the integrated > ParameterTool.fromPropertiesFile function of Flink. > > Locally this version works absolutely fine: > val configuration = > ParameterTool.fromPropertiesFile("src/main/resources/config.properties") > > But on the cluster this didn't work, so we introduced this version, which > also doesn't work: > > val baseParameters = > ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile) > > gives > > java.io.FileNotFoundException: Properties file > file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties > does not exist > > The property file is located in src/main/resources. > Do you have any idea of how to integrate such property files into the jar > package? > > -Thanks > Simon |
Re: Import Configuration File in Flink Cluster
|
In reply to this post by simon peyer
Are you using Maven to package your project? I believe the resources plugin[1] can suit your needs.
[1]: http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html On Mon, May 23, 2016 at 3:56 PM, simon peyer <[hidden email]> wrote:
BR, Stefano Baghino |
Re: Import Configuration File in Flink Cluster
|
|
Hi @Max So for each file in the src/main/resources folder, I first have to create a new file, copy the file from the resources folder to this new file and then I'm able to parse it? @Stefano I think the files in src/main/resources are integrated automatically right? Or am I missing something? Cheers Simon
On 23 May 2016, at 16:28, Maximilian Michels <[hidden email]> wrote: Hi Simon, You'll have to write the property file to disk first to load it using the ParameterTool.fromPropertiesFile method. For example: // copy config from Java resource to a file File configOnDisk = new File("/path/to/config.properties"); Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"), configOnDisk.toPath()); // load the new file ParameterTool.fromPropertiesFile(configOnDisk); Cheers, Max
|
Re: Import Configuration File in Flink Cluster
|
|
Hi Simon,
AFAIK this is the way to go. We could add a method to the ParameterTool which loads from a resource to make it more convenient. Cheers, Max On Mon, May 23, 2016 at 4:42 PM, simon peyer <[hidden email]> wrote: > Hi > > @Max > So for each file in the src/main/resources folder, I first have to create a > new file, copy the file from the resources folder to this new file and then > I'm able to parse it? > > @Stefano > I think the files in src/main/resources are integrated automatically right? > Or am I missing something? > > Cheers > Simon > > > On 23 May 2016, at 16:30, Stefano Baghino <[hidden email]> > wrote: > > Are you using Maven to package your project? I believe the resources > plugin[1] can suit your needs. > > [1]: > http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html > > > On 23 May 2016, at 16:28, Maximilian Michels <[hidden email]> wrote: > > Hi Simon, > > You'll have to write the property file to disk first to load it using > the ParameterTool.fromPropertiesFile method. > > For example: > > // copy config from Java resource to a file > File configOnDisk = new File("/path/to/config.properties"); > Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"), > configOnDisk.toPath()); > // load the new file > ParameterTool.fromPropertiesFile(configOnDisk); > > > Cheers, > Max > > > > On Mon, May 23, 2016 at 3:56 PM, simon peyer <[hidden email]> wrote: >> >> Hi together >> >> Currently I'm using flink on a docker cluster on AWS. >> I would like to use property files with the integrated >> ParameterTool.fromPropertiesFile function of Flink. >> >> Locally this version works absolutely fine: >> val configuration = >> ParameterTool.fromPropertiesFile("src/main/resources/config.properties") >> >> But on the cluster this didn't work, so we introduced this version, which >> also doesn't work: >> >> val baseParameters = >> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile) >> >> gives >> >> java.io.FileNotFoundException: Properties file >> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties >> does not exist >> >> The property file is located in src/main/resources. >> Do you have any idea of how to integrate such property files into the jar >> package? >> >> -Thanks >> Simon > > > > > -- > BR, > Stefano Baghino > > Software Engineer @ Radicalbit > > |
Re: Import Configuration File in Flink Cluster
|
|
Hi Max
Thanks a lot for your helpful answer. It now works on the cluster. It would be great to have a method for loading from resources. -Cheers Simon > On 23 May 2016, at 17:52, Maximilian Michels <[hidden email]> wrote: > > Hi Simon, > > AFAIK this is the way to go. We could add a method to the > ParameterTool which loads from a resource to make it more convenient. > > Cheers, > Max > > On Mon, May 23, 2016 at 4:42 PM, simon peyer <[hidden email]> wrote: >> Hi >> >> @Max >> So for each file in the src/main/resources folder, I first have to create a >> new file, copy the file from the resources folder to this new file and then >> I'm able to parse it? >> >> @Stefano >> I think the files in src/main/resources are integrated automatically right? >> Or am I missing something? >> >> Cheers >> Simon >> >> >> On 23 May 2016, at 16:30, Stefano Baghino <[hidden email]> >> wrote: >> >> Are you using Maven to package your project? I believe the resources >> plugin[1] can suit your needs. >> >> [1]: >> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html >> >> >> On 23 May 2016, at 16:28, Maximilian Michels <[hidden email]> wrote: >> >> Hi Simon, >> >> You'll have to write the property file to disk first to load it using >> the ParameterTool.fromPropertiesFile method. >> >> For example: >> >> // copy config from Java resource to a file >> File configOnDisk = new File("/path/to/config.properties"); >> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"), >> configOnDisk.toPath()); >> // load the new file >> ParameterTool.fromPropertiesFile(configOnDisk); >> >> >> Cheers, >> Max >> >> >> >> On Mon, May 23, 2016 at 3:56 PM, simon peyer <[hidden email]> wrote: >>> >>> Hi together >>> >>> Currently I'm using flink on a docker cluster on AWS. >>> I would like to use property files with the integrated >>> ParameterTool.fromPropertiesFile function of Flink. >>> >>> Locally this version works absolutely fine: >>> val configuration = >>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties") >>> >>> But on the cluster this didn't work, so we introduced this version, which >>> also doesn't work: >>> >>> val baseParameters = >>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile) >>> >>> gives >>> >>> java.io.FileNotFoundException: Properties file >>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties >>> does not exist >>> >>> The property file is located in src/main/resources. >>> Do you have any idea of how to integrate such property files into the jar >>> package? >>> >>> -Thanks >>> Simon >> >> >> >> >> -- >> BR, >> Stefano Baghino >> >> Software Engineer @ Radicalbit >> >> |
Re: Import Configuration File in Flink Cluster
|
|
I was gonna post the exact question and noticed this thread.
It will be great if we can have a method in parameter tool to load from resources. Thanks Simon :) Abhinav Bajaj Senior Engineer HERE Predictive Analytics Office: +12062092767 Mobile: +17083299516 HERE Seattle 701 Pike Street, #2000, Seattle, WA 98101, USA 47° 36' 41" N. 122° 19' 57" W HERE Maps On 5/23/16, 8:54 AM, "simon peyer" <[hidden email]> wrote: >Hi Max > >Thanks a lot for your helpful answer. >It now works on the cluster. >It would be great to have a method for loading from resources. > >-Cheers >Simon > > >> On 23 May 2016, at 17:52, Maximilian Michels <[hidden email]> wrote: >> >> Hi Simon, >> >> AFAIK this is the way to go. We could add a method to the >> ParameterTool which loads from a resource to make it more convenient. >> >> Cheers, >> Max >> >> On Mon, May 23, 2016 at 4:42 PM, simon peyer <[hidden email]> wrote: >>> Hi >>> >>> @Max >>> So for each file in the src/main/resources folder, I first have to create a >>> new file, copy the file from the resources folder to this new file and then >>> I'm able to parse it? >>> >>> @Stefano >>> I think the files in src/main/resources are integrated automatically right? >>> Or am I missing something? >>> >>> Cheers >>> Simon >>> >>> >>> On 23 May 2016, at 16:30, Stefano Baghino <[hidden email]> >>> wrote: >>> >>> Are you using Maven to package your project? I believe the resources >>> plugin[1] can suit your needs. >>> >>> [1]: >>> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html >>> >>> >>> On 23 May 2016, at 16:28, Maximilian Michels <[hidden email]> wrote: >>> >>> Hi Simon, >>> >>> You'll have to write the property file to disk first to load it using >>> the ParameterTool.fromPropertiesFile method. >>> >>> For example: >>> >>> // copy config from Java resource to a file >>> File configOnDisk = new File("/path/to/config.properties"); >>> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"), >>> configOnDisk.toPath()); >>> // load the new file >>> ParameterTool.fromPropertiesFile(configOnDisk); >>> >>> >>> Cheers, >>> Max >>> >>> >>> >>> On Mon, May 23, 2016 at 3:56 PM, simon peyer <[hidden email]> wrote: >>>> >>>> Hi together >>>> >>>> Currently I'm using flink on a docker cluster on AWS. >>>> I would like to use property files with the integrated >>>> ParameterTool.fromPropertiesFile function of Flink. >>>> >>>> Locally this version works absolutely fine: >>>> val configuration = >>>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties") >>>> >>>> But on the cluster this didn't work, so we introduced this version, which >>>> also doesn't work: >>>> >>>> val baseParameters = >>>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile) >>>> >>>> gives >>>> >>>> java.io.FileNotFoundException: Properties file >>>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties >>>> does not exist >>>> >>>> The property file is located in src/main/resources. >>>> Do you have any idea of how to integrate such property files into the jar >>>> package? >>>> >>>> -Thanks >>>> Simon >>> >>> >>> >>> >>> -- >>> BR, >>> Stefano Baghino >>> >>> Software Engineer @ Radicalbit >>> >>> > |
Re: Import Configuration File in Flink Cluster
|
|
Hi I just want to let you know, that I did implement a version for wrapping properties in the cluster: object PropertiesWrapper { def getFromConfig(configFile: String): ParameterTool = { val prop: Properties = new Properties() var input: InputStream = null try { input = PropertiesWrapper.getClass.getResourceAsStream("/" + configFile); // load a properties file prop.load(input) } finally { if (input != null) { input.close(); } } ParameterTool.fromMap(prop.asInstanceOf[Map[String,String]]) } def mergeArgsWithProperties(configuration:Properties, args: Array[String]):Properties = { configuration } } Have a nice day --Simon
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |