Hi guys,
I get java heap space error when I have 1 GB of TM memory and 4 slots(we have 4 cores in our lower environment) per TM , each slot has 1/4GB of managed memory. From the flink doc https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html#task-slots-and-resources I see that slots are allocated memory statically . When I change the TM memory to 8GB , my job works fine without any heap issues with 4 slots. So here each slot gets around 2GB of heap . In another environment we have 60 cores. Does it make sense for me to have 60 slots in the task manager for 8GB of TM heap ? I assume that I will get heap space error since each slot will have 8/60 GB of memory . Is my assumption correct ? Thanks, Vishwas |
Hi Vishwas, First of all, 8 GB for 60 cores is not a lot. You might not be able to utilize all cores when running Flink. However, the memory usage depends on several things. Assuming your are using Flink for stream processing, the type of the state backend is important. If you use the FSStateBackend, all state is stored on the JVM heap which is certainly not want you want. However, also the RocksDBStateBackend which stores data on the hard disk has memory requirements. IIRC a few hundred MBs per keyed state and task. So 60 tasks with keyed state might not possible due to the memory requirements of RocksDB. In addition to the state backend, you need to configure a sufficient number of network buffers. These also depend on the number of slots that the TaskManager process offers. Besides the state backend and network buffers, there are some fixed memory costs per TaskManager process. A slot itself (without any user code running) does not take much memory. Note that Flink does not enforce any memory budgets for slots. Every slot can take as much JVM memory as it wants. However, if there's not enough memory, the whole TaskManager crashes with an OutOfMemoryError. This is not a definitive answer to your question, but hopefully helps you to configure the cluster for your use case. Best, Fabian Am Di., 1. Okt. 2019 um 20:59 Uhr schrieb Vishwas Siravara <[hidden email]>:
|
Free forum by Nabble | Edit this page |