Correct way to package application.properties file with Flink JAR

[Manas Kale]

Hi,

I have a Flink job that I am packaging as a JAR that is submitted to the Flink cluster runtime. However, this JAR reads a few configuration values from a .properties file.

What is the recommended way to package this properties file when submitting to a cluster? Do I have to copy it to a folder in my flink cluster installation?

My own attempt is a somewhat convoluted method that is not working. Basically I set an environment variable that points to the properties file, and I use that at runtime to read configuration values. This works when I run it in my IDE as a minicluster but fails when I submit it to the cluster. I'm kind of stuck debugging this as for some reason I am not able to see the logs from the configuration reader class (asked a question about that in a separate thread).

[Chesnay Schepler]

You could bundle said file in the jar and retrieve it via
getClass().getClassLoader().getResource("<file name>").

On 10/21/2020 2:24 PM, Manas Kale wrote:

> Hi,
> I have a Flink job that I am packaging as a JAR that is submitted to
> the Flink cluster runtime. However, this JAR reads a few configuration
> values from a .properties file.
> What is the recommended way to package this properties file when
> submitting to a cluster? Do I have to copy it to a folder in my flink
> cluster installation?
>
> My own attempt is a somewhat convoluted method that is not working.
> Basically I set an environment variable that points to the properties
> file, and I use that at runtime to read configuration values. This
> works when I run it in my IDE as a minicluster but fails when I submit
> it to the cluster. I'm kind of stuck debugging this as for some reason
> I am not able to see the logs from the configuration reader class
> (asked a question about that in a separate thread).

[Manas Kale]

Hi Chesnay,

The Config reader has everything static, so I tried using 

Config.class.getClassLoader().getResourceAsStream("pipeline.properties");

Here's the .pom file for that file:

<build>

   <resources>
      <resource>
         <directory>src/main/resources</directory>
         <includes>
            <include>pipeline.properties</include>
            <include>log4j.properties</include>
         </includes>
      </resource>
   </resources>

...

</build>

I can see the pipeline.properties packaged in my JAR at the root level (using maven package command).

However, this works on my IDE minicluster but loads null values when submitted to the cluster.

The Config class is not at the package root, rather it is a few levels deep. Does that make a difference and cause the getClassLoader() to treat an inner package as root?

On Wed, Oct 21, 2020 at 6:06 PM Chesnay Schepler <[hidden email]> wrote:

You could bundle said file in the jar and retrieve it via
getClass().getClassLoader().getResource("<file name>").

On 10/21/2020 2:24 PM, Manas Kale wrote:
> Hi,
> I have a Flink job that I am packaging as a JAR that is submitted to
> the Flink cluster runtime. However, this JAR reads a few configuration
> values from a .properties file.
> What is the recommended way to package this properties file when
> submitting to a cluster? Do I have to copy it to a folder in my flink
> cluster installation?
>
> My own attempt is a somewhat convoluted method that is not working.
> Basically I set an environment variable that points to the properties
> file, and I use that at runtime to read configuration values. This
> works when I run it in my IDE as a minicluster but fails when I submit
> it to the cluster. I'm kind of stuck debugging this as for some reason
> I am not able to see the logs from the configuration reader class
> (asked a question about that in a separate thread).

[Manas Kale]

Okay, I solved the other issue with viewing logs which proved that correct, non-null values are being loaded. I believe I have a different issue altogether so will create a separate thread for that. Thanks for the help Chesnay!

On Thu, Oct 22, 2020 at 11:30 AM Manas Kale <[hidden email]> wrote:

Hi Chesnay,

The Config reader has everything static, so I tried using 

Config.class.getClassLoader().getResourceAsStream("pipeline.properties");

Here's the .pom file for that file:

<build>

   <resources>
      <resource>
         <directory>src/main/resources</directory>
         <includes>
            <include>pipeline.properties</include>
            <include>log4j.properties</include>
         </includes>
      </resource>
   </resources>

...

</build>

I can see the pipeline.properties packaged in my JAR at the root level (using maven package command).

However, this works on my IDE minicluster but loads null values when submitted to the cluster.

The Config class is not at the package root, rather it is a few levels deep. Does that make a difference and cause the getClassLoader() to treat an inner package as root?

On Wed, Oct 21, 2020 at 6:06 PM Chesnay Schepler <[hidden email]> wrote:

You could bundle said file in the jar and retrieve it via
getClass().getClassLoader().getResource("<file name>").

On 10/21/2020 2:24 PM, Manas Kale wrote:
> Hi,
> I have a Flink job that I am packaging as a JAR that is submitted to
> the Flink cluster runtime. However, this JAR reads a few configuration
> values from a .properties file.
> What is the recommended way to package this properties file when
> submitting to a cluster? Do I have to copy it to a folder in my flink
> cluster installation?
>
> My own attempt is a somewhat convoluted method that is not working.
> Basically I set an environment variable that points to the properties
> file, and I use that at runtime to read configuration values. This
> works when I run it in my IDE as a minicluster but fails when I submit
> it to the cluster. I'm kind of stuck debugging this as for some reason
> I am not able to see the logs from the configuration reader class
> (asked a question about that in a separate thread).