Flink Application cluster/standalone job: some JVM Options added to Program Arguments

[Alexey Trenikhun]

Hello,

I was trying to deploy Flink 1.12.0 Application cluster on k8s, I have following job manager arguments:

standalone-job --job-classname com.x.App --job-id 00000000000000000000000000000000 @/opt/flink/conf/fsp.conf

However, when I print args from App.main():

[@/opt/flink/conf/ssp.conf, -D, jobmanager.memory.off-heap.size=134217728b, -D, jobmanager.memory.jvm-overhead.min=280552338b, -D, jobmanager.memory.jvm-metaspace.size=268435456b, -D, jobmanager.memory.heap.size=2122317824b, -D, jobmanager.memory.jvm-overhead.max=280552338b]

Looks like 

With Flink 1.11.3 same deployment worked as expected - the job was only getting [@/opt/flink/conf/ssp.conf]

Thanks,

Alexey

[Matthias]

Hi Alexey,

thanks for reaching out to the Flink community. I'm not 100% sure whether you have an actual issue or whether it's just the changed behavior you are confused about. The change you're describing was introduced in Flink 1.12 as part of the work on FLIP-104 [1] exposing the actual memory usage through the web UI.

Does this answer your question?

Best,
Matthias

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager

On Sat, Jan 16, 2021 at 5:35 AM Alexey Trenikhun <[hidden email]> wrote:

Hello,

I was trying to deploy Flink 1.12.0 Application cluster on k8s, I have following job manager arguments:

standalone-job --job-classname com.x.App --job-id 00000000000000000000000000000000 @/opt/flink/conf/fsp.conf

However, when I print args from App.main():

[@/opt/flink/conf/ssp.conf, -D, jobmanager.memory.off-heap.size=134217728b, -D, jobmanager.memory.jvm-overhead.min=280552338b, -D, jobmanager.memory.jvm-metaspace.size=268435456b, -D, jobmanager.memory.heap.size=2122317824b, -D, jobmanager.memory.jvm-overhead.max=280552338b]

Looks like 

With Flink 1.11.3 same deployment worked as expected - the job was only getting [@/opt/flink/conf/ssp.conf]

Thanks,

Alexey

[Alexey Trenikhun]

Hi Matthias,

As I understand FLIP-104 is about UI. My question is about what user job is supposed to do with arguments like this: “-Djobmanager.memory.off-heap.size=134217728b“, set system property ? If user code has nothing to do with such arguments, why Flink append these arguments to user JOB args?

Thanks,

Alexey

---

From: Matthias Pohl <[hidden email]>
Sent: Sunday, January 17, 2021 11:53:29 PM
To: Alexey Trenikhun <[hidden email]>
Cc: Flink User Mail List <[hidden email]>
Subject: Re: Flink Application cluster/standalone job: some JVM Options added to Program Arguments

 

Hi Alexey,

thanks for reaching out to the Flink community. I'm not 100% sure whether you have an actual issue or whether it's just the changed behavior you are confused about. The change you're describing was introduced in Flink 1.12 as part of the work on FLIP-104 [1] exposing the actual memory usage through the web UI.

Does this answer your question?

Best,
Matthias

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager

On Sat, Jan 16, 2021 at 5:35 AM Alexey Trenikhun <[hidden email]> wrote:

Hello,

I was trying to deploy Flink 1.12.0 Application cluster on k8s, I have following job manager arguments:

standalone-job --job-classname com.x.App --job-id 00000000000000000000000000000000 @/opt/flink/conf/fsp.conf

However, when I print args from App.main():

[@/opt/flink/conf/ssp.conf, -D, jobmanager.memory.off-heap.size=134217728b, -D, jobmanager.memory.jvm-overhead.min=280552338b, -D, jobmanager.memory.jvm-metaspace.size=268435456b, -D, jobmanager.memory.heap.size=2122317824b, -D, jobmanager.memory.jvm-overhead.max=280552338b]

Looks like 

With Flink 1.11.3 same deployment worked as expected - the job was only getting [@/opt/flink/conf/ssp.conf]

Thanks,

Alexey

[Matthias]

You're right. Thinking about it and looking through the code, I agree: The dynamic properties shouldn't be exposed in the main method. I was able to reproduce the described behavior. I created FLINK-21024 covering this.

Thanks for reporting this issue, Alexey.

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-21024

On Mon, Jan 18, 2021 at 7:11 PM Alexey Trenikhun <[hidden email]> wrote:

Hi Matthias,

As I understand FLIP-104 is about UI. My question is about what user job is supposed to do with arguments like this: “-Djobmanager.memory.off-heap.size=134217728b“, set system property ? If user code has nothing to do with such arguments, why Flink append these arguments to user JOB args?

Thanks,

Alexey

---

From: Matthias Pohl <[hidden email]>
Sent: Sunday, January 17, 2021 11:53:29 PM
To: Alexey Trenikhun <[hidden email]>
Cc: Flink User Mail List <[hidden email]>
Subject: Re: Flink Application cluster/standalone job: some JVM Options added to Program Arguments

 

Hi Alexey,

thanks for reaching out to the Flink community. I'm not 100% sure whether you have an actual issue or whether it's just the changed behavior you are confused about. The change you're describing was introduced in Flink 1.12 as part of the work on FLIP-104 [1] exposing the actual memory usage through the web UI.

Does this answer your question?

Best,
Matthias

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-104%3A+Add+More+Metrics+to+Jobmanager

On Sat, Jan 16, 2021 at 5:35 AM Alexey Trenikhun <[hidden email]> wrote:

Hello,

I was trying to deploy Flink 1.12.0 Application cluster on k8s, I have following job manager arguments:

standalone-job --job-classname com.x.App --job-id 00000000000000000000000000000000 @/opt/flink/conf/fsp.conf

However, when I print args from App.main():

[@/opt/flink/conf/ssp.conf, -D, jobmanager.memory.off-heap.size=134217728b, -D, jobmanager.memory.jvm-overhead.min=280552338b, -D, jobmanager.memory.jvm-metaspace.size=268435456b, -D, jobmanager.memory.heap.size=2122317824b, -D, jobmanager.memory.jvm-overhead.max=280552338b]

Looks like 

With Flink 1.11.3 same deployment worked as expected - the job was only getting [@/opt/flink/conf/ssp.conf]

Thanks,

Alexey