Does Flink use all of the available memory?

[Viktor Rosenfeld]

Hi,

does the default configuration of Flink use all of the available memory? I mean the physical memory installed in the computer, not whatever amount of memory the JVM allocates by default.

Cheers,
Viktor

[Fabian Hueske-2]

Hi,

Flink uses only the memory which is configured to the JobManager and TaskManager JVMs.

By default this is 256MB for the JM and 512MB for the TM (see [1] for details).

The TM memory is split into equally large chunks for each configured slot of the TM.

You should definitely configure the TM JVM memory for your setup! See common options in [1] for other important configuration parameters such as numberOfTaskSlots.

In Yarn setups the TM memory is automatically sized with respect to the Yarn container size, if I remember correctly.

[1] http://flink.apache.org/docs/0.8/config.html

2015-02-14 13:58 GMT+01:00 Viktor Rosenfeld <[hidden email]>:

Hi,

does the default configuration of Flink use all of the available memory? I
mean the physical memory installed in the computer, not whatever amount of
memory the JVM allocates by default.

Cheers,
Viktor



--
View this message in context: http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Does-Flink-use-all-of-the-available-memory-tp730.html
Sent from the Apache Flink (Incubator) User Mailing List archive. mailing list archive at Nabble.com.

[Viktor Rosenfeld]

Hi Fabian,

thanks for the reply. I increased the TM heap to 20GB (on a 32GB machine) and indeed got a performance boost of more than a minute (about 20%).

Cheers,

Viktor

Am 14.02.2015 um 14:23 schrieb Fabian Hueske-2 [via Apache Flink (Incubator) User Mailing List archive.] <[hidden email]>:

Hi,

Flink uses only the memory which is configured to the JobManager and TaskManager JVMs.

By default this is 256MB for the JM and 512MB for the TM (see [1] for details).

The TM memory is split into equally large chunks for each configured slot of the TM.

You should definitely configure the TM JVM memory for your setup! See common options in [1] for other important configuration parameters such as numberOfTaskSlots.

In Yarn setups the TM memory is automatically sized with respect to the Yarn container size, if I remember correctly.

[1] http://flink.apache.org/docs/0.8/config.html

2015-02-14 13:58 GMT+01:00 Viktor Rosenfeld <<a href="x-msg://43/user/SendEmail.jtp?type=node&amp;node=731&amp;i=0" target="_top" rel="nofollow" link="external" class="">[hidden email]>:

Hi,

does the default configuration of Flink use all of the available memory? I
mean the physical memory installed in the computer, not whatever amount of
memory the JVM allocates by default.

Cheers,
Viktor



--
View this message in context: http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Does-Flink-use-all-of-the-available-memory-tp730.html
Sent from the Apache Flink (Incubator) User Mailing List archive. mailing list archive at Nabble.com.

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Does-Flink-use-all-of-the-available-memory-tp730p731.html

To unsubscribe from Does Flink use all of the available memory?, click here.
NAML