(DEPRECATED) Apache Flink User Mailing List archive.

Re: passing additional jvm parameters to the configuration

Posted by Georg Heiler on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/passing-additional-jvm-parameters-to-the-configuration-tp36200p36220.html

Hi,

but how can I change/configure it per submitted job and not for the whole cluster?

Best,
Georg

Am Do., 25. Juni 2020 um 10:07 Uhr schrieb Arvid Heise <[hidden email]>:
Hi Georg,

thank you for your detailed explanation. You want to use env.java.opts[1]. There are flavors if you only want to make it available on job manager or task manager but I guess the basic form is good enough for you.


On Wed, Jun 24, 2020 at 10:52 PM Georg Heiler <[hidden email]> wrote:
Hi Arvid,

thanks for the quick reply. I have a strong Apache spark background. There, when executing on YARN or locally usually, the cluster is created on-demand for the duration of the batch /streaming job.
There, there is only the concept of A) master/driver (application master) B) slave/executor C) Driver: the node where the main class is invoked. In Sparks`notion, I want the -D parameter to be available on the (C) Driver node. When translating this to Flink, I want this to be available to the Main class which is invoked when the job is submitted/started by the job manager (which should be equivalent to the driver).

But maybe my understanding of Flink is not 100% correct yet.

Unfortunately, using -D directly is not working.

Best,
Georg

Am Mi., 24. Juni 2020 um 22:13 Uhr schrieb Arvid Heise <[hidden email]>:
Hi Georg,

could you check if simply using -D is working as described here [1].

If not, could you please be more precise: do you want the parameter to be passed to the driver, the job manager, or the task managers?


On Wed, Jun 24, 2020 at 8:55 PM Georg Heiler <[hidden email]> wrote:
Hi,

how can I pass additional configuration parameters like spark`s extraJavaOptions to a flink job?


contains the details. But the gist is:
flink run --class com.github.geoheil.streamingreference.tweets.TweetsAnalysis \
"usecases/tweets/build/libs/tweets_${SCALA_VERSION}-${VERSION}-all.jar" \
-yD env.java.opts="-Dconfig.file='config/jobs/twitter-analysis.conf'"

is not passing the -Dconfig.file to the flink job!

Best,
Georg


--

Arvid Heise | Senior Java Developer


Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng   


--

Arvid Heise | Senior Java Developer


Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng   
Free forum by Nabble Edit this page