Visualize result of Flink job
Locked
 
|
Hi there
I am using Flink to analyse a lot of incoming data. Every 10 seconds it makes sense to present the analysis so far as some form of visualization. Every 10 seconds I therefore will replace the current contents of the visualization/presentation with the analysis result of the most recent 10 seconds. I was first thinking of using ElasticSearch/Kibana for this because I know it should be easy to set up, but I am thinking it may not be the best fit, because Elastic is by nature a search engine that is good for trending and stuff like that - not entire replacement of the current view. And therefore I may also experience difficulties implementing the view in Elastic. Does anyone know of any other visualization tools that work well with Flink? ...where it is easy to export the result of a Flink job to a user interface (web). Thanks Palle |
Re: Visualize result of Flink job
|
|
> I am thinking it may not be the best fit, because Elastic is by nature a search engine that is good for trending and stuff like that - not entire replacement of the current view.
Why u think that the elasticsearch is not the right tool? To visualise something u have to find what to visualise first, right? On Sun, May 29, 2016 at 8:20 PM, Palle <[hidden email]> wrote: > Hi there > > I am using Flink to analyse a lot of incoming data. Every 10 seconds it makes sense to present the analysis so far as some form of visualization. Every 10 seconds I therefore will replace the current contents of the visualization/presentation with the analysis result of the most recent 10 seconds. > > I was first thinking of using ElasticSearch/Kibana for this because I know it should be easy to set up, but I am thinking it may not be the best fit, because Elastic is by nature a search engine that is good for trending and stuff like that - not entire replacement of the current view. And therefore I may also experience difficulties implementing the view in Elastic. > > Does anyone know of any other visualization tools that work well with Flink? ...where it is easy to export the result of a Flink job to a user interface (web). > > Thanks > Palle |
|
|
I know exactly what to visualize. As I wrote, it is the latest result of the Flink job I would like to visualize. There is no need to use elastic to find it first.
The data I have is of such a nature that they every 10 seconds could be written into a file, meaning that the file at all times would contain the most recent results (at latest 10 seconds old). I am not interested in the history, and therefore I should think elastic is not the best fit. So my question is if anyone knows of a component (Apache or other) that can make the visualization a little nicer than just the file :-) ----- Original meddelelse ----- > Fra: Kanstantsin Kamkou <[hidden email]> > Til: [hidden email] > Dato: Søn, 29. maj 2016 22:42 > Emne: Re: Visualize result of Flink job > > > I am thinking it may not be the best fit, because Elastic is by > nature a search engine that is good for trending and stuff like that > - not entire replacement of the current view. > Why u think that the elasticsearch is not the right tool? To > visualise > something u have to find what to visualise first, right? > > > On Sun, May 29, 2016 at 8:20 PM, Palle <[hidden email]> wrote: > > Hi there > > > > I am using Flink to analyse a lot of incoming data. Every 10 > seconds it makes sense to present the analysis so far as some form > of visualization. Every 10 seconds I therefore will replace the > current contents of the visualization/presentation with the analysis > result of the most recent 10 seconds. > > > > I was first thinking of using ElasticSearch/Kibana for this > because I know it should be easy to set up, but I am thinking it may > not be the best fit, because Elastic is by nature a search engine > that is good for trending and stuff like that - not entire > replacement of the current view. And therefore I may also experience > difficulties implementing the view in Elastic. > > > > Does anyone know of any other visualization tools that work well > with Flink? ...where it is easy to export the result of a Flink job > to a user interface (web). > > > > Thanks > > Palle |
|
|
OK, I found product that seems to be what I am looking for: Apache Zeppelin. I will have a look into that one. If anyone can point me to an example (Git) outputting data from Flink to the Zeppelin Notebook I would be happy.
----- Original meddelelse ----- > Fra: Palle <[hidden email]> > Til: [hidden email] > Dato: Man, 30. maj 2016 08:20 > Emne: Re: Visualize result of Flink job > > I know exactly what to visualize. As I wrote, it is the latest > result of the Flink job I would like to visualize. There is no need > to use elastic to find it first. > > The data I have is of such a nature that they every 10 seconds > could be written into a file, meaning that the file at all times > would contain the most recent results (at latest 10 seconds old). I > am not interested in the history, and therefore I should think > elastic is not the best fit. So my question is if anyone knows of a > component (Apache or other) that can make the visualization a little > nicer than just the file :-) > > > ----- Original meddelelse ----- > > Fra: Kanstantsin Kamkou <[hidden email]> > > Til: [hidden email] > > Dato: Søn, 29. maj 2016 22:42 > > Emne: Re: Visualize result of Flink job > > > > > I am thinking it may not be the best fit, because Elastic is by > > nature a search engine that is good for trending and stuff like > that > > - not entire replacement of the current view. > > Why u think that the elasticsearch is not the right tool? To > > visualise > > something u have to find what to visualise first, right? > > > > > > On Sun, May 29, 2016 at 8:20 PM, Palle <[hidden email]> wrote: > > > Hi there > > > > > > I am using Flink to analyse a lot of incoming data. Every 10 > > seconds it makes sense to present the analysis so far as some form > > of visualization. Every 10 seconds I therefore will replace the > > current contents of the visualization/presentation with the > analysis > > result of the most recent 10 seconds. > > > > > > I was first thinking of using ElasticSearch/Kibana for this > > because I know it should be easy to set up, but I am thinking it > may > > not be the best fit, because Elastic is by nature a search engine > > that is good for trending and stuff like that - not entire > > replacement of the current view. And therefore I may also > experience > > difficulties implementing the view in Elastic. > > > > > > Does anyone know of any other visualization tools that work well > > with Flink? ...where it is easy to export the result of a Flink > job > > to a user interface (web). > > > > > > Thanks > > > Palle > |
|
|
Hi Palle, I think there is currently no way of sending the data from a streaming Flink job into Zeppelin. What rate / amount of data do you expect to send every 10 seconds to the visualization tool? People have used Flink -> ES -> Kibana for this purpose in the past [1], but I think you can not send millions of records per second into ES. Something like 1000 - 5000 elements / second should easily work for a small ES setup. On Mon, May 30, 2016 at 10:00 AM, Palle <[hidden email]> wrote: OK, I found product that seems to be what I am looking for: Apache Zeppelin. I will have a look into that one. If anyone can point me to an example (Git) outputting data from Flink to the Zeppelin Notebook I would be happy. |
|
|
Hi Robert.
Thank you for the answer. I am looking at a rate of max 10.000 elements / 10 seconds, so Elastic/Kibana is probably the way to go. I'll find a way to model it. Thanks. /Palle ----- Original meddelelse ----- Fra: Robert Metzger <[hidden email]> |
|
|
Hi, sounds doable. I think it should be easy to set up a first proof of concept for this. Let us know if you need any further assistance. Regards, Robert On Mon, May 30, 2016 at 2:29 PM, Palle <[hidden email]> wrote:
|
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |