Executing graph algorithms on Gelly that are larger then memmory

[otherwise777]

I read somewhere that Flink and Gelly should be able to handle graph algorithms that require more space then the available memory, i'm currently getting java OutOfMemoryError heap space and if it would use disk space that wouldn't happen.
Currently my algorithms use dense graphs with 10m edges, the algorithms that i use are on my github [1], i did get the same heap space errors when using the Gelly algorithms, i also tried playing with setting the parallelism but it mostly gives the same error

Can anyone help me with some methods on how to solve this so that i can scale up to bigger graphs?

[1] https://github.com/otherwise777/Temporal_Graph_library/

[otherwise777]

Small addition, i'm currently running the programs via my IDE intelij

[Vasiliki Kalavri]

Hi,

can you give us some more details about the algorithm you are testing and your configuration?

Flink DataSet operators like join, coGroup, reduce, etc. spill to disk if there is not enough memory. If you are using a delta iteration operator though, the state that is kept across iterations (solution set) must fit in memory.

-Vasia.

On 28 November 2016 at 21:44, otherwise777 <[hidden email]> wrote:

Small addition, i'm currently running the programs via my IDE intelij



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Executing-graph-algorithms-on-Gelly-that-are-larger-then-memmory-tp10358p10359.html

Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.