usae of ClusterSpecificationBuilder.taskManagerMemoryMB

[Vitaliy Semochkin]

Hi,

what ClusterSpecificationBuilder.taskManagerMemoryMB is for in flink 1.10?

It's only usage I see is in YarnCluserDescriptor.validateClusterResources and I do not get the meaning of it.

How is it different from taskmanager.memory.process.size?

And what's the point of having it, if it's not used anywhere?

Regards,

Vitaliy

[Xintong Song]

Hi Vitality,

After FLIP-49, ClusterSpecification.taskManagerMemoryMB is no longer necessary. It can be completely replaced by `taskmanager.memory.process.size`. It is kept merely for legacy reasons.

I'm actually thinking about removing ClusterSpecification, maybe after finishing FLIP-116 [1], which replaces masterMemoryMB with `jobmanager.memory.process.size`. That would also involve refactoring YarnClusterDescriptor, which is not in good shape (e.g. the method startAppMaster has more than 400 lines) and is closely coupled with ClusterSpecification.

On Tue, Mar 24, 2020 at 5:59 AM Vitaliy Semochkin <[hidden email]> wrote:

Hi,

what ClusterSpecificationBuilder.taskManagerMemoryMB is for in flink 1.10?

It's only usage I see is in YarnCluserDescriptor.validateClusterResources and I do not get the meaning of it.

How is it different from taskmanager.memory.process.size?

And what's the point of having it, if it's not used anywhere?

Regards,

Vitaliy

[Vitaliy Semochkin]

Got it, thank you very much for the reply.

So far we can not avoid using ClusterSpecification because

 clusterDescriptor.deployJobCluster(clusterSpecification, jobGraph... ) depends on it.

Best Regards,

Vitaliy

On Tue, Mar 24, 2020 at 5:24 AM Xintong Song <[hidden email]> wrote:

Hi Vitality,

After FLIP-49, ClusterSpecification.taskManagerMemoryMB is no longer necessary. It can be completely replaced by `taskmanager.memory.process.size`. It is kept merely for legacy reasons.

I'm actually thinking about removing ClusterSpecification, maybe after finishing FLIP-116 [1], which replaces masterMemoryMB with `jobmanager.memory.process.size`. That would also involve refactoring YarnClusterDescriptor, which is not in good shape (e.g. the method startAppMaster has more than 400 lines) and is closely coupled with ClusterSpecification.

Thank you~

Xintong Song

On Tue, Mar 24, 2020 at 5:59 AM Vitaliy Semochkin <[hidden email]> wrote:

Hi,

what ClusterSpecificationBuilder.taskManagerMemoryMB is for in flink 1.10?

It's only usage I see is in YarnCluserDescriptor.validateClusterResources and I do not get the meaning of it.

How is it different from taskmanager.memory.process.size?

And what's the point of having it, if it's not used anywhere?

Regards,

Vitaliy