Passing application configuring to Flink uber jar

[Krishna Kalyan]

Hello All,

I have a Flink application that inherits configuration from application.conf in the resources folder. 

Now, I want to run this application on a cluster. I begin creating an fat jar "mvn clean assembly". 

This jar file is executed by executing the following command below

$FLINK_HOME/bin/flink run --class "com.test...." -D "env.java.opts=-Dconfig.file=dev.conf" /opt/resources/iot-flink-assembly-0.1.0.jar

This causes and error that the configuration key was not found. I could someone please let me know how to pass custom application.conf environment file to cluster. I could not find a lot of information online on how to do this.

Regards,

Krishna

Reference:

[1] https://stackoverflow.com/questions/46157479/apache-flink-how-do-i-specify-an-external-application-conf-file?rq=1

[2] https://github.com/lightbend/config/issues/84

--

Standorte in Stuttgart und Berlin · Zoi TechCon GmbH · Quellenstr. 7 · 70376 Stuttgart · Geschäftsführer: Benjamin Hermann, Dr. Daniel Heubach. Amtsgericht Stuttgart HRB 759619, Gerichtsstand Stuttgart. Die genannten Angaben werden automatisch hinzugefügt und lassen keine Rückschlüsse auf den Rechtscharakter der E-Mail zu. This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.

[Till Rohrmann]

Hi Krishna,

I think the problem is that you are trying to pass in dynamic properties (-Dconfig.file=dev.conf) to an already started the cluster. The Flink cluster components or their JVMs need to know the env.java.opts at cluster start up time and not when the Flink job is submitted. You can check this by looking into the logs. They contain the parameters with which the JVMs are started.

You could, however, use the per job mode where you start a cluster per job (-m yarn-cluster).

Cheers,

Till

On Thu, Nov 22, 2018 at 3:27 PM Krishna Kalyan <[hidden email]> wrote:

Hello All,

I have a Flink application that inherits configuration from application.conf in the resources folder. 

Now, I want to run this application on a cluster. I begin creating an fat jar "mvn clean assembly". 

This jar file is executed by executing the following command below

$FLINK_HOME/bin/flink run --class "com.test...." -D "env.java.opts=-Dconfig.file=dev.conf" /opt/resources/iot-flink-assembly-0.1.0.jar

This causes and error that the configuration key was not found. I could someone please let me know how to pass custom application.conf environment file to cluster. I could not find a lot of information online on how to do this.

Regards,

Krishna

Reference:

[1] https://stackoverflow.com/questions/46157479/apache-flink-how-do-i-specify-an-external-application-conf-file?rq=1

[2] https://github.com/lightbend/config/issues/84

--

Standorte in Stuttgart und Berlin · Zoi TechCon GmbH · Quellenstr. 7 · 70376 Stuttgart · Geschäftsführer: Benjamin Hermann, Dr. Daniel Heubach. Amtsgericht Stuttgart HRB 759619, Gerichtsstand Stuttgart. Die genannten Angaben werden automatisch hinzugefügt und lassen keine Rückschlüsse auf den Rechtscharakter der E-Mail zu. This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.