How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

[Tony Wei]

Hi,

I put the my configuration file in `./src/main/resources/` and packed it inside my jar.

I want to run it on standalone cluster by using web UI to submit my job.

No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.

Is there any best practice to deal with such problem? Thanks for your help.

Best Regards,

Tony Wei

[Aljoscha Krettek]

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei

[Tony Wei]

Hi Aljoscha,

I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.

Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf

Best Regards,

Tony Wei

2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei

[Tony Wei]

Hi Aljoscha,

I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar.

So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.

I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.

Best Regards,

Tony Wei

2017-09-08 23:40 GMT+08:00 Tony Wei <[hidden email]>:

Hi Aljoscha,

I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.

Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf

Best Regards,

Tony Wei

2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei

[Aljoscha Krettek]

Hi,

Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.

Best,

Aljoscha

On 11. Sep 2017, at 04:46, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar.

So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.

I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.

Best Regards,

Tony Wei

2017-09-08 23:40 GMT+08:00 Tony Wei <[hidden email]>:

Hi Aljoscha,

I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.

Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf

Best Regards,

Tony Wei

2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei

[Aljoscha Krettek]

Hi,

I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to ParameterTool.fromPropertiesFile(). Passing a File should work because a File can be constructed from an URI and a URL can be turned into a URI.

Would you be interested in opening a Jira issue for that and working on it?

Best,

Aljoscha

On 15. Sep 2017, at 03:32, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I used Maven with this command "mvn clean package -Pbuild-jar" to create the jar.

Best Regards,

Tony Wei

Aljoscha Krettek <[hidden email]>於 2017年9月14日 週四，下午6:24寫道：

Hi,

Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.

Best,

Aljoscha

On 11. Sep 2017, at 04:46, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar.

So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.

I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.

Best Regards,

Tony Wei

2017-09-08 23:40 GMT+08:00 Tony Wei <[hidden email]>:

Hi Aljoscha,

I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.

Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf

Best Regards,

Tony Wei

2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei

[Tony Wei]

Hi Aljoscha,

Thanks for your reply. It looks great to have hat feature. I will create a Jira issue for that and try to solve it.

Best Regards,

Tony Wei

2017-09-15 20:51 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to ParameterTool.fromPropertiesFile(). Passing a File should work because a File can be constructed from an URI and a URL can be turned into a URI.

Would you be interested in opening a Jira issue for that and working on it?

Best,

Aljoscha

On 15. Sep 2017, at 03:32, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I used Maven with this command "mvn clean package -Pbuild-jar" to create the jar.

Best Regards,

Tony Wei

Aljoscha Krettek <[hidden email]>於 2017年9月14日 週四，下午6:24寫道：

Hi,

Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.

Best,

Aljoscha

On 11. Sep 2017, at 04:46, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar.

So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.

I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.

Best Regards,

Tony Wei

2017-09-08 23:40 GMT+08:00 Tony Wei <[hidden email]>:

Hi Aljoscha,

I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.

Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf

Best Regards,

Tony Wei

2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei

[Ron Crocker]

What’s crazy is that I just stumbled on the same issue. Thanks for sharing!

Ron

—

Ron Crocker

Principal Engineer & Architect

( ( •)) New Relic

[hidden email]

M: [phone]

On Sep 15, 2017, at 7:30 AM, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

Thanks for your reply. It looks great to have hat feature. I will create a Jira issue for that and try to solve it.

Best Regards,

Tony Wei

2017-09-15 20:51 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to ParameterTool.fromPropertiesFile(). Passing a File should work because a File can be constructed from an URI and a URL can be turned into a URI.

Would you be interested in opening a Jira issue for that and working on it?

Best,

Aljoscha

On 15. Sep 2017, at 03:32, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I used Maven with this command "mvn clean package -Pbuild-jar" to create the jar.

Best Regards,

Tony Wei

Aljoscha Krettek <[hidden email]>於 2017年9月14日 週四，下午6:24寫道：

Hi,

Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.

Best,

Aljoscha

On 11. Sep 2017, at 04:46, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar.

So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.

I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.

Best Regards,

Tony Wei

2017-09-08 23:40 GMT+08:00 Tony Wei <[hidden email]>:

Hi Aljoscha,

I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.

Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf

Best Regards,

Tony Wei

2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei

[Aljoscha Krettek]

Hi Ron,

This is the issue and it was already implemented for upcoming 1.3.3 and 1.4.0.

https://issues.apache.org/jira/browse/FLINK-7630

Best,

Aljoscha

On 26. Sep 2017, at 21:10, Ron Crocker <[hidden email]> wrote:

What’s crazy is that I just stumbled on the same issue. Thanks for sharing!

Ron

—

Ron Crocker

Principal Engineer & Architect

( ( •)) New Relic

[hidden email]

M: [phone]

On Sep 15, 2017, at 7:30 AM, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

Thanks for your reply. It looks great to have hat feature. I will create a Jira issue for that and try to solve it.

Best Regards,

Tony Wei

2017-09-15 20:51 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

I think calling getPath() on the URL returned from getResource() loses some of the information that is required to resolve the file in the jar. The solution should be to allow passing a "File" to ParameterTool.fromPropertiesFile() or to allow passing an InputStream to ParameterTool.fromPropertiesFile(). Passing a File should work because a File can be constructed from an URI and a URL can be turned into a URI.

Would you be interested in opening a Jira issue for that and working on it?

Best,

Aljoscha

On 15. Sep 2017, at 03:32, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I used Maven with this command "mvn clean package -Pbuild-jar" to create the jar.

Best Regards,

Tony Wei

Aljoscha Krettek <[hidden email]>於 2017年9月14日 週四，下午6:24寫道：

Hi,

Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE.

Best,

Aljoscha

On 11. Sep 2017, at 04:46, Tony Wei <[hidden email]> wrote:

Hi Aljoscha,

I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar.

So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format and used ParameterTool.fromArgs() to parse them with other arguments.

I'm not sure if this is a good solution. If you have any better one, please let me know. Thanks for your help.

Best Regards,

Tony Wei

2017-09-08 23:40 GMT+08:00 Tony Wei <[hidden email]>:

Hi Aljoscha,

I have tried `StreamJob.class.getClassLoader().getResource("application.conf").getPath()`, but I got this exception.

Caused by: java.io.FileNotFoundException: Properties file /home/tonywei/flink/file:/tmp/flink-web-24351e69-a261-45be-9503-087db8155a8f/d69a3ca9-bfa0-43ef-83e8-e15f38162a87_quickstart-0.1.jar!/application.conf

Best Regards,

Tony Wei

2017-09-08 23:24 GMT+08:00 Aljoscha Krettek <[hidden email]>:

Hi,

How are you specifying the path for the properties file? Have you tried reading the properties by using this.getClass().getClassLoader().getResource()?

Best,
Aljoscha

> On 8. Sep 2017, at 16:32, Tony Wei <[hidden email]> wrote:
>
> Hi,
>
> I put the my configuration file in `./src/main/resources/` and packed it inside my jar.
> I want to run it on standalone cluster by using web UI to submit my job.
> No matter which way I tried, the ParameterTool.fromPropertiesFile() couldn't find the file path, but threw `FileNotFoundException` instead.
> Is there any best practice to deal with such problem? Thanks for your help.
>
> Best Regards,
> Tony Wei