How to do code isolation if muiltple jobs run on the same taskmanager process?

[Lei Wang]

Consider the following situation.

Two jobs builed in the same jar, they will share some common code, for example, some static constants variables. 

Currently they are running on the same task manager process.

I  killed job1, changed the static variable to another and then submit it again.

Does another job will get the new value of the static variable or still use the old value?

Thanks,
Lei  

[Arvid Heise-4]

Hi Lei,

each application has its own classloader as such each static constant exists multiple times (1 per job). So there should be no interference. You could verify it by logging the value of the constant and see it yourself.

Best,

Arvid

On Thu, Mar 11, 2021 at 7:11 AM Lei Wang <[hidden email]> wrote:

Consider the following situation.

Two jobs builed in the same jar, they will share some common code, for example, some static constants variables. 

Currently they are running on the same task manager process.

I  killed job1, changed the static variable to another and then submit it again.

Does another job will get the new value of the static variable or still use the old value?

Thanks,
Lei  

[Lei Wang]

Thanks Arvid, 

If too many jobs run in the same task manager JVM,  will it  cause too much metaspace memory occupation? 

  

Thanks,
Lei 

On Thu, Mar 11, 2021 at 11:54 PM Arvid Heise <[hidden email]> wrote:

Hi Lei,

each application has its own classloader as such each static constant exists multiple times (1 per job). So there should be no interference. You could verify it by logging the value of the constant and see it yourself.

Best,

Arvid

On Thu, Mar 11, 2021 at 7:11 AM Lei Wang <[hidden email]> wrote:

Consider the following situation.

Two jobs builed in the same jar, they will share some common code, for example, some static constants variables. 

Currently they are running on the same task manager process.

I  killed job1, changed the static variable to another and then submit it again.

Does another job will get the new value of the static variable or still use the old value?

Thanks,
Lei  

[Arvid Heise-4]

Hi Lei,

yes, metaspace would run out eventually if you run too much in parallel. All finished jobs will close the classloaders and free the metaspace memory.

For newer setups, we recommend creating an ad-hoc cluster for each Flink application for this and several other reasons. If you are already using K8s, I'd definitively switch to per-application clusters.

You could also have a look at the Ververica community editions for easy managing of such applications. [1]

[1] https://www.ververica.com/getting-started

On Fri, Mar 12, 2021 at 4:15 AM Lei Wang <[hidden email]> wrote:

Thanks Arvid, 

If too many jobs run in the same task manager JVM,  will it  cause too much metaspace memory occupation? 

  

Thanks,
Lei 

On Thu, Mar 11, 2021 at 11:54 PM Arvid Heise <[hidden email]> wrote:

Hi Lei,

each application has its own classloader as such each static constant exists multiple times (1 per job). So there should be no interference. You could verify it by logging the value of the constant and see it yourself.

Best,

Arvid

On Thu, Mar 11, 2021 at 7:11 AM Lei Wang <[hidden email]> wrote:

Consider the following situation.

Two jobs builed in the same jar, they will share some common code, for example, some static constants variables. 

Currently they are running on the same task manager process.

I  killed job1, changed the static variable to another and then submit it again.

Does another job will get the new value of the static variable or still use the old value?

Thanks,
Lei