Slow startup for slaves with much memory

Posted by Arvid Heise on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Slow-startup-for-slaves-with-much-memory-tp25.html

Hi Flinker,

no question - just brief feedback.

If you have lots of memory available (TM_HEAP=.7TB), the startup time of slaves can be quite cumbersome.

1) -Xms parameter in the taskmanager.sh takes a long time to initialize the JVM. No log file is avail in that time and .out is empty. It took me quite a while to figure out what was wrong (in fact nothing, but it looked as if the taskmanagers crashed).

2) After removing that parameter, a second bottleneck occurs. The memory manager initializes the memory upfront. Therefore, the task manager is still not registered and tasks cannot be scheduled. (it takes up to 30 minutes for .5TB to be initalized)

I'm now running the experiments with 10% of the available RAM. Not an ideal solution, however, my environment is somewhat special.

Best,

Arvid