Re: Slow startup for slaves with much memory

Posted by Arvid Heise on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Slow-startup-for-slaves-with-much-memory-tp25p29.html

You are right with Xms, it's going much faster, but the startup of the JVM is slower (up to 5 min).

Still to speed up my cold cache experiments, I'm now using way less memory for memory manager.

On Wed, Aug 13, 2014 at 4:30 PM, Stephan Ewen <[hidden email]> wrote:

We are doing lazy memory initialization in the next versions. Nevertheless, it seems a bit hard that the JVM takes 30 minutes just to gather 700 GB of byte arrays.

Can you make sure that Xms and Xmx are the same? Otherwise, the heap space grows incrementally with tenured garbage collections, which takes longer.

On Wed, Aug 13, 2014 at 2:36 PM, Arvid Heise <[hidden email]> wrote:

Hi Flinker,

no question - just brief feedback.

If you have lots of memory available (TM_HEAP=.7TB), the startup time of slaves can be quite cumbersome.

1) -Xms parameter in the taskmanager.sh takes a long time to initialize the JVM. No log file is avail in that time and .out is empty. It took me quite a while to figure out what was wrong (in fact nothing, but it looked as if the taskmanagers crashed).

2) After removing that parameter, a second bottleneck occurs. The memory manager initializes the memory upfront. Therefore, the task manager is still not registered and tasks cannot be scheduled. (it takes up to 30 minutes for .5TB to be initalized)

I'm now running the experiments with 10% of the available RAM. Not an ideal solution, however, my environment is somewhat special.

Best,

Arvid