Re: How large a Flink cluster can have?

Posted by Alexis Gendronneau on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/How-large-a-Flink-cluster-can-have-tp7949p8661.html

Hi,

If possible, with this information, that be great to know  how jobmanager has to be scaled according to number of nodes ? 

Will 1 Jobmanager be able to manage hundreds of nodes ? Is there recommandation in terms of JM/TM ratio ? 

Thanks 

2016-07-14 15:41 GMT+02:00 Robert Metzger <[hidden email]>:

Hi,

I think the reason why this information is not written anywhere is because we don't know it either.

Alibaba seems to run a fork of Flink on "thousands of nodes" [1].

Maybe some of the production users on this mailing list can share some information regarding this.

[1] http://www.slideshare.net/HadoopSummit/improvements-to-flink-its-applications-in-alibaba-search

On Wed, Jul 13, 2016 at 3:54 PM, Yan Chou Chen <[hidden email]> wrote:

FAQ[1], mailing list[2], and the powered by page[3] doesn't find
related information. Just out of curiosity, what is the current
largest Flink cluster size running in production? For instance, long
time ago yahoo [4] ran 4k hadoop nodes in production.

Thanks

[1]. https://flink.apache.org/faq.html
[2]. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=search_page&node=1&query=max+cluster&i=48
[3]. https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink
[4]. https://developer.yahoo.com/blogs/hadoop/scaling-hadoop-4000-nodes-yahoo-410.html

--