Providing external files to flink classpath

[Vishwas Siravara]

Hi ,
I am trying to add external property files to the flink classpath for
my application. These files are not a part of the fat jar. I put them
under the lib folder but flink cant find them? How can I manage
external property files that needs to be read by flink ?

Thanks,
Vishwas

[Yun Tang]

Hi Vishwas

1. You could use '-yt' to ship specified files to the class path, please refer to [1] for more details.
2. If the properties are only loaded on client side before executing the application, you could let your application to just read from local property data. Flink support to load properties within the ParameterTool [2].
   

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/cli.html#usage

[2] https://github.com/apache/flink/blob/f1721293b0701d584d42bd68671181e332d2ad04/flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java#L120

Best

Yun Tang

---

From: Vishwas Siravara <[hidden email]>
Sent: Saturday, June 29, 2019 0:43
To: user
Subject: Providing external files to flink classpath

 

Hi ,
I am trying to add external property files to the flink classpath for
my application. These files are not a part of the fat jar. I put them
under the lib folder but flink cant find them? How can I manage
external property files that needs to be read by flink ?

Thanks,
Vishwas

[Vishwas Siravara]

Does the -yt option work for standalone cluster without dedicated resource manager ? So this property file is read by one of the dependencies inside my application as a file, so I can't really use Parameter tool to parse the config file. 

Thanks,

Vishwas 

On Fri, Jun 28, 2019 at 11:08 PM Yun Tang <[hidden email]> wrote:

Hi Vishwas

1. You could use '-yt' to ship specified files to the class path, please refer to [1] for more details.
2. If the properties are only loaded on client side before executing the application, you could let your application to just read from local property data. Flink support to load properties within the ParameterTool [2].
   

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/cli.html#usage

[2] https://github.com/apache/flink/blob/f1721293b0701d584d42bd68671181e332d2ad04/flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java#L120

Best

Yun Tang

---

From: Vishwas Siravara <[hidden email]>
Sent: Saturday, June 29, 2019 0:43
To: user
Subject: Providing external files to flink classpath

 

Hi ,
I am trying to add external property files to the flink classpath for
my application. These files are not a part of the fat jar. I put them
under the lib folder but flink cant find them? How can I manage
external property files that needs to be read by flink ?

Thanks,
Vishwas

[Maxim Parkachov]

Hi Vishwas,

took me some time to find out as well. If you have your properties file under lib following will work:

val kafkaPropertiesInputStream = getClass.getClassLoader.getResourceAsStream("lib/config/kafka.properties")

Hope this helps,

Maxim.

On Wed, Jul 17, 2019 at 7:23 PM Vishwas Siravara <[hidden email]> wrote:

Does the -yt option work for standalone cluster without dedicated resource manager ? So this property file is read by one of the dependencies inside my application as a file, so I can't really use Parameter tool to parse the config file. 

Thanks,

Vishwas 

On Fri, Jun 28, 2019 at 11:08 PM Yun Tang <[hidden email]> wrote:

Hi Vishwas

1. You could use '-yt' to ship specified files to the class path, please refer to [1] for more details.
2. If the properties are only loaded on client side before executing the application, you could let your application to just read from local property data. Flink support to load properties within the ParameterTool [2].
   

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/cli.html#usage

[2] https://github.com/apache/flink/blob/f1721293b0701d584d42bd68671181e332d2ad04/flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java#L120

Best

Yun Tang

---

From: Vishwas Siravara <[hidden email]>
Sent: Saturday, June 29, 2019 0:43
To: user
Subject: Providing external files to flink classpath

 

Hi ,
I am trying to add external property files to the flink classpath for
my application. These files are not a part of the fat jar. I put them
under the lib folder but flink cant find them? How can I manage
external property files that needs to be read by flink ?

Thanks,
Vishwas