Graph with stream of updates

[Ankur Sharma]

Hello,

Is it possible to create and update graph with streaming edge and vertex data in flink?

Best,

Ankur Sharma

3.15 E1.1 Universität des Saarlandes

66123, Saarbrücken Germany

Email: [hidden email] 

            [hidden email]

[rmetzger0]

Hi Ankur,

Can you provide a bit more information on what you are trying to achieve?

Do you want to keep a graph build from an stream of events within Flink and query that?

Or you you want to change the dataflow graph of Flink while a job is running?

Regards,

Robert

On Thu, Feb 25, 2016 at 11:19 PM, Ankur Sharma <[hidden email]> wrote:

Hello,

Is it possible to create and update graph with streaming edge and vertex data in flink?

Best,

Ankur Sharma

3.15 E1.1 Universität des Saarlandes

66123, Saarbrücken Germany

Email: [hidden email] 

            [hidden email]

[Ankur Sharma]

Hello,

Thanks for reply.

I want to create a graph from stream and query it. You got it right.

Stream may be edges that are getting added or removed from the graph.

Is there a way to create a empty global graph that can be transformed using a stream of updates?

Best,

Ankur Sharma

3.15 E1.1 Universität des Saarlandes

66123, Saarbrücken Germany

Email: [hidden email] 

            [hidden email]

On 26 Feb 2016, at 14:55, Robert Metzger <[hidden email]> wrote:

Hi Ankur,

Can you provide a bit more information on what you are trying to achieve?

Do you want to keep a graph build from an stream of events within Flink and query that?

Or you you want to change the dataflow graph of Flink while a job is running?

Regards,

Robert

On Thu, Feb 25, 2016 at 11:19 PM, Ankur Sharma <[hidden email]> wrote:

Hello,

Is it possible to create and update graph with streaming edge and vertex data in flink?

Best,

Ankur Sharma

3.15 E1.1 Universität des Saarlandes

66123, Saarbrücken Germany

Email: [hidden email] 

            [hidden email]

[Vasiliki Kalavri]

Hi Ankur,

you can have custom state in your Flink operators, including a graph. There is no graph state abstraction provided at the moment, but it shouldn't be too hard for you to implement your own.

If your use-case only requires processing edge additions only, then you might want to take a look into gelly-stream [1]. Is it a single-pass graph streaming API, processing edge additions, and operating on graph summaries.

Cheers,

-Vasia.

[1]: https://github.com/vasia/gelly-streaming

On 26 February 2016 at 14:59, Ankur Sharma <[hidden email]> wrote:

Hello,

Thanks for reply.

I want to create a graph from stream and query it. You got it right.

Stream may be edges that are getting added or removed from the graph.

Is there a way to create a empty global graph that can be transformed using a stream of updates?

Best,

Ankur Sharma

3.15 E1.1 Universität des Saarlandes

66123, Saarbrücken Germany

Email: [hidden email] 

            [hidden email]

On 26 Feb 2016, at 14:55, Robert Metzger <[hidden email]> wrote:

Hi Ankur,

Can you provide a bit more information on what you are trying to achieve?

Do you want to keep a graph build from an stream of events within Flink and query that?

Or you you want to change the dataflow graph of Flink while a job is running?

Regards,

Robert

On Thu, Feb 25, 2016 at 11:19 PM, Ankur Sharma <[hidden email]> wrote:

Hello,

Is it possible to create and update graph with streaming edge and vertex data in flink?

Best,

Ankur Sharma

3.15 E1.1 Universität des Saarlandes

66123, Saarbrücken Germany

Email: [hidden email] 

            [hidden email]

[agentmilindu]

Hi Vasia and Ankur,

I have the same need as Ankur where I want to create a graph from Twitter stream and query it. I got the streaming graph built from Twitter stream thanks to your Gelly Stream. Now I want to run queries( different graph algorithms ) on this streaming graph time to time, while it keeps building over the time independently.  I'm new to Flink, please help me with this.

Thank you,
Milindu.

[Vasiliki Kalavri]

Hi Milindu,

as far as I know, there is currently no way to query the state from outside of Flink. That's a feature in the roadmap, but I'm not sure when it will be provided. Maybe someone else can give us an update.

For now, you can either implement your queries inside you streaming job and output a result from time to time, e.g. using a time window (or slice() in gelly-stream) or you could periodically dump your graph to a database like neo4j and run your queries there.

Cheers,

-Vasia.

On 6 July 2016 at 08:07, agentmilindu <[hidden email]> wrote:

Hi Vasia and Ankur,

I have the same need as Ankur where I want to create a graph from Twitter
stream and query it. I got the streaming graph built from Twitter stream
thanks to your Gelly Stream. Now I want to run queries( different graph
algorithms ) on this streaming graph time to time, while it keeps building
over the time independently.  I'm new to Flink, please help me with this.

Thank you,
Milindu.



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Graph-with-stream-of-updates-tp5166p7837.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.