Distribute crawling of a URL list using Flink

Posted by Eranga Heshan on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Distribute-crawling-of-a-URL-list-using-Flink-tp14873.html

Hi all,

I am fairly new to Flink. I have this project where I have a list of URLs (In one node) which need to be crawled distributedly. Then for each URL, I need the serialized crawled result to be written to a single text file.

I want to know if there are similar projects which I can look into or an idea on how to implement this.

Thanks & Regards,