Re: Flink first project

Posted by Georg Heiler on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Flink-first-project-tp12754p12771.html

So you would suggest flume over a custom akka-source from bahir?

Jörn Franke <[hidden email]> schrieb am So., 23. Apr. 2017 um 18:59 Uhr:

I would use flume to import these sources to HDFS and then use flink or Hadoop or whatever to process them. While it is possible to do it in flink, you do not want that your processing fails because the web service is not available etc.

Via flume which is suitable for this kind of tasks it is more controlled and reliable.

On 23. Apr 2017, at 18:02, Georg Heiler <[hidden email]> wrote:

New to flink I would like to do a small project to get a better feeling for flink. I am thinking of getting some stats from several REST api (i.e. Bitcoin course values from different exchanges) and comparing prices over different exchanges in real time.

Are there already some REST api sources for flink as a sample to get started implementing a custom REST source?

I was thinking about using https://github.com/timmolter/XChange to connect to several exchanges. E.g. to make a single api call by hand would look similar to

val currencyPair = new CurrencyPair(Currency.XMR, Currency.BTC)

  CertHelper.trustAllCerts()

  val poloniex = ExchangeFactory.INSTANCE.createExchange(classOf[PoloniexExchange].getName)

  val dataService = poloniex.getMarketDataService

  generic(dataService)

  raw(dataService.asInstanceOf[PoloniexMarketDataServiceRaw])

System.out.println(dataService.getTicker(currencyPair))

How would be a proper way to make this available as a flink source? I have seen https://github.com/apache/flink/blob/master/flink-contrib/flink-connector-wikiedits/src/main/java/org/apache/flink/streaming/connectors/wikiedits/WikipediaEditsSource.java but new to flink am a bit unsure how to proceed.

Regards,

Georg