Should I decrease the taskmanager.memory.fraction ?

[Renkai]

I've just read the article https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html, It seems that the MemorySegment mainly used for sort and join for batch execute. But I mainly use Flink for streaming work, I do a lot of map/flatMap and reduce in Flink. Is the MemorySegment still useful for my application?If yes how could the streaming applications benefit from it?

[Stephan Ewen]

Streaming applications currently do not benefit from this, which will hopefully change soon.

Until then, you need not do anything, because the memory is lazily allocated, and will never be allocated for streaming jobs.

On Tue, Dec 6, 2016 at 8:59 AM, Renkai <[hidden email]> wrote:

I've just read the article
https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-and-Bytes.html,
It seems that the MemorySegment mainly used for sort and join for batch
execute. But I mainly use Flink for streaming work, I do a lot of
map/flatMap and reduce in Flink. Is the MemorySegment still useful for my
application?If yes how could the streaming applications benefit from it?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Should-I-decrease-the-taskmanager-memory-fraction-tp10475.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[nragon]

Hi,

Regarding this topic, does flink uses memory segments for window states?
Is memory managment only "used at is full capacity" for batch?

Thanks

[Fabian Hueske-2]

Yes, that's correct. Managed memory is currently only used for batch processing.

For stream processing data is either kept on the JVM heap (InMemoryStateBackend, FsStateBackend) or in RocksDB (RocksDBStateBackend).
There are plans to implement a state backend that serializes to to managed memory (possibly spilling to disk if memory is exceeded).

2017-04-06 16:36 GMT+02:00 nragon <[hidden email]>:

Hi,

Regarding this topic, does flink uses memory segments for window states?
Is memory managment only "used at is full capacity" for batch?

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Should-I-decrease-the-taskmanager-memory-fraction-tp10475p12540.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[nragon]

Like stephan said we don't need to worry about memory managment with streaming? Meaning if I have 3 taskmanagers with 2Gb each, memory segments won't be pre-allocated and I will have 2Gb heap for streaming?

Thanks

[Fabian Hueske-2]

Yes, by default managed memory is lazily allocated. Since streaming applications don't request management memory, it won't be allocated

2017-04-06 16:52 GMT+02:00 nragon <[hidden email]>:

Like stephan said we don't need to worry about memory managment with
streaming? Meaning if I have 3 taskmanagers with 2Gb each, memory segments
won't be pre-allocated and I will have 2Gb heap for streaming?

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Should-I-decrease-the-taskmanager-memory-fraction-tp10475p12542.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[nragon]

Ok, got it :)
But since memory segments are good to avoid gc problems, will streaming jobs suffer from gc when keeping window or any other states?