Source code question - about the logic of calculating network buffer

[徐涛]

Hi Experts,

I am debugging the WordCount Flink streaming program in local mode. Flink version is 1.7.2

I saw the following calculation logic about network buffer in class TaskManagerServices.

jvmHeapNoNet is equal to -xmx amount in Java.

why the networkBufBytes = xmx / (1-networkBufFraction) * networkBufFraction, it is a bit strange.

why it is not calculated by xmx * networkBufFraction?

 

final long networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
   (long) (jvmHeapNoNet / (1.0 - networkBufFraction) * networkBufFraction)));

Best 

Henry

[Yun Gao]

Hi tao,

      As a whole, `networkBufBytes` is not part of the heap. In fact, it is allocated from the direct memory. The rough relationship (ignores min/max and assumes managed memory is allocated on heap) between the variables are:

        

              Total memory of TM (configured by taskmanager.heap.mb, the name may be a little misleading) = heap memory (-XmX) + networkBufBytes.

              networkBufBytes = Total memory of TM * networkBufFraction.

     Therefore, the networkBufBytes = xmx / (1 - networkBufFraction) * networkBufFraction.

Best,

Yun

------------------------------------------------------------------

From:徐涛 <[hidden email]>

Send Time:2019 Jun. 13 (Thu.) 10:23

To:user <[hidden email]>

Subject:Source code question - about the logic of calculating network buffer

Hi Experts,

I am debugging the WordCount Flink streaming program in local mode. Flink version is 1.7.2

I saw the following calculation logic about network buffer in class TaskManagerServices.

jvmHeapNoNet is equal to -xmx amount in Java.

why the networkBufBytes = xmx / (1-networkBufFraction) * networkBufFraction, it is a bit strange.

why it is not calculated by xmx * networkBufFraction?

 

final long networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
   (long) (jvmHeapNoNet / (1.0 - networkBufFraction) * networkBufFraction)));

Best 

Henry

[徐涛]

Hi Yun,

Thanks a lot for the detailed and clear explanation, that is very helpful.

Best

Henry

在 2019年6月13日，上午10:32，Yun Gao <[hidden email]> 写道：

Hi tao,

      As a whole, `networkBufBytes` is not part of the heap. In fact, it is allocated from the direct memory. The rough relationship (ignores min/max and assumes managed memory is allocated on heap) between the variables are:

        

              Total memory of TM (configured by taskmanager.heap.mb, the name may be a little misleading) = heap memory (-XmX) + networkBufBytes.

              networkBufBytes = Total memory of TM * networkBufFraction.

     Therefore, the networkBufBytes = xmx / (1 - networkBufFraction) * networkBufFraction.

Best,

Yun

------------------------------------------------------------------

From:徐涛 <[hidden email]>

Send Time:2019 Jun. 13 (Thu.) 10:23

To:user <[hidden email]>

Subject:Source code question - about the logic of calculating network buffer

Hi Experts,

I am debugging the WordCount Flink streaming program in local mode. Flink version is 1.7.2

I saw the following calculation logic about network buffer in class TaskManagerServices.

jvmHeapNoNet is equal to -xmx amount in Java.

why the networkBufBytes = xmx / (1-networkBufFraction) * networkBufFraction, it is a bit strange.

why it is not calculated by xmx * networkBufFraction?

 

final long networkBufBytes = Math.min(networkBufMax, Math.max(networkBufMin,
   (long) (jvmHeapNoNet / (1.0 - networkBufFraction) * networkBufFraction)));

Best 

Henry