Re: Looking for relevant sources related to connecting Apache Flink and Edgent.

Posted by Kostas Kloudas on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Looking-for-relevant-sources-related-to-connecting-Apache-Flink-and-Edgent-tp24817p24825.html

Hi again,

I forgot to say that, unfortunately, I am not familiar with Apache Edgent, but if you can write your filter in Edgent's programming model,

Then you can push your data from Edgent to a third party storage system (e.g. Kafka, HDFS, etc) and use Flink's connectors, instead of 

having to implement a custom source.

Cheers,
Kostas

On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas <[hidden email]> wrote:

Hi Felipe,

This seems related to your previous question about a custom scheduler that knows which task to run on which machine.

As Chesnay said, this is a rather involved and laborious task, if you want to do it as a general framework.

But if you know what operation to push down, then why not decoupling the two and implementing the filtering as a separate job 

running on your Raspberry and a new job which consumes the output of the first and does the analytics?

Cheers,

Kostas

On Thu, Nov 29, 2018 at 10:23 AM Felipe Gutierrez <[hidden email]> wrote:

Hi,

I am trying to design a little prototype with Flink and Apache Edgent (http://edgent.apache.org/) and I would like some help on the direction for it. I am running Flink at my laptop and Edgent on my Raspberry Pi with a simple filter for a proximity sensor (https://github.com/felipegutierrez/explore-rpi/blob/master/src/main/java/org/sense/edgent/app/UltrasonicEdgentApp.java).

My idea is to push down the filter operator from Flink to the Raspberry Pi which is running Apache Edgent. With this in mind, where do you guys advise me to start?

I have some ideas to study...

1 - Try to get the list of operators that Flink is about to execute on the JobManager. source: https://ci.apache.org/projects/flink/flink-docs-stable/internals/job_scheduling.html

2 - Implement a connector to Apache Edgent in order to exchange messages between Flink and Edgent.

Do you guys think in another source that is interesting regarding my prototype?

Thanks,

Felipe