Creating Graphs from DataStream in Flink Gelly

[UFUOMA IGHOROJE]

Is there some support for creating Graphs from DataStream in Flink Gelly? My use case is to create dynamic graphs, where the graph topology can be updated from a stream of data. From the API, all I can find is creating graphs from DataSets.

Best,

Ufuoma

[Andra Lungu]

Hi,

There is a separate project related to graph streaming. It's called gelly-streaming.

And, if you look here:
https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/GraphStream.java

You can find a constructor which creates a graph from a DataStream of edges.

Gelly itself cannot do that, for now.

Hope this helps!

Andra

On Mon, Nov 2, 2015 at 3:35 PM, UFUOMA IGHOROJE <[hidden email]> wrote:

Is there some support for creating Graphs from DataStream in Flink Gelly? My use case is to create dynamic graphs, where the graph topology can be updated from a stream of data. From the API, all I can find is creating graphs from DataSets.

Best,

Ufuoma

[Vasiliki Kalavri]

Hi Ufuoma,

Gelly doesn't support dynamic streaming graphs yet.

The project Andra has linked to is a prototype for *one-pass* streaming graph analytics, i.e. no graph state is maintained.

If you would like to keep and maintain the graph state in your streaming program, you would have to implement it yourself.

Take a look at the relevant section of the programming guide [1] and please let us know if you have questions!

Cheers,

-Vasia.

[1]: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#working-with-state

On 2 November 2015 at 14:39, Andra Lungu <[hidden email]> wrote:

Hi,

There is a separate project related to graph streaming. It's called gelly-streaming.

And, if you look here:
https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/GraphStream.java

You can find a constructor which creates a graph from a DataStream of edges.

Gelly itself cannot do that, for now.

Hope this helps!

Andra

On Mon, Nov 2, 2015 at 3:35 PM, UFUOMA IGHOROJE <[hidden email]> wrote:

Is there some support for creating Graphs from DataStream in Flink Gelly? My use case is to create dynamic graphs, where the graph topology can be updated from a stream of data. From the API, all I can find is creating graphs from DataSets.

Best,

Ufuoma

[UFUOMA IGHOROJE]

Hi Vasia & Andra, 

Thanks for the pointers. I guess I have to implement basic support for dynamic streaming graphs. I’ll share my solutions or issues as I go on.

Best,

Ufuoma 

On 02 Nov 2015, at 17:51, Vasiliki Kalavri <[hidden email]> wrote:

Hi Ufuoma,

Gelly doesn't support dynamic streaming graphs yet.

The project Andra has linked to is a prototype for *one-pass* streaming graph analytics, i.e. no graph state is maintained.

If you would like to keep and maintain the graph state in your streaming program, you would have to implement it yourself.

Take a look at the relevant section of the programming guide [1] and please let us know if you have questions!

Cheers,

-Vasia.

[1]: https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#working-with-state

On 2 November 2015 at 14:39, Andra Lungu <[hidden email]> wrote:

Hi,

There is a separate project related to graph streaming. It's called gelly-streaming.

And, if you look here:
https://github.com/vasia/gelly-streaming/blob/master/src/main/java/org/apache/flink/graph/streaming/GraphStream.java

You can find a constructor which creates a graph from a DataStream of edges.

Gelly itself cannot do that, for now.

Hope this helps!

Andra

On Mon, Nov 2, 2015 at 3:35 PM, UFUOMA IGHOROJE <[hidden email]> wrote:

Is there some support for creating Graphs from DataStream in Flink Gelly? My use case is to create dynamic graphs, where the graph topology can be updated from a stream of data. From the API, all I can find is creating graphs from DataSets.

Best,

Ufuoma