Hundreds of parallel jobs on Flink Cluster

[chiggi_dev]

Hi,

I am coming across a use case where I may have to run more than100 parallel jobs(which may have different processing needs) on a Flink cluster. 

My flink cluster, currently, has 1 Job Manager and 4/5 Task Managers depending on the processing needed is running on a Kubernetes cluster with 3 worker nodes.   

I have few questions around this before I start looking at how I can execute this use case on such cluster configuration : 

1) Is there any limitation w.r.t  Job Manager while running 100 parallel jobs on a similar cluster? 

2) Is there any limitation on the number of jobs that can be run on a cluster with 1 Job Manager? 

3) Are there any recommendations around having hundreds of jobs on a single cluster?

4) Would I need multiple clusters for such a use case or are there any tuning parameters which can help for eg. Akka tuning parameters? 

Thanks in advance,

Chirag  

[Till Rohrmann]

Hi Chirag,

if you give your Flink cluster enough resources, especially slots, to execute all of the 100 parallel jobs, then it should not be a big problem. A Flink session cluster can run multiple jobs in parallel as long as you it has enough slots to deploy the task to.

There are some tuning parameters but it is hard to say which one of them would apply to your use case since it heavily depends on your jobs (e.g. size of operators, checkpointing settings, etc.). I would suggest that you deploy the jobs to your Flink cluster and then report back if you should stumble across any problems.

Cheers,

Till

On Fri, Jun 8, 2018 at 7:49 AM Chirag Dewan <[hidden email]> wrote:

Hi,

I am coming across a use case where I may have to run more than100 parallel jobs(which may have different processing needs) on a Flink cluster. 

My flink cluster, currently, has 1 Job Manager and 4/5 Task Managers depending on the processing needed is running on a Kubernetes cluster with 3 worker nodes.   

I have few questions around this before I start looking at how I can execute this use case on such cluster configuration : 

1) Is there any limitation w.r.t  Job Manager while running 100 parallel jobs on a similar cluster? 

2) Is there any limitation on the number of jobs that can be run on a cluster with 1 Job Manager? 

3) Are there any recommendations around having hundreds of jobs on a single cluster?

4) Would I need multiple clusters for such a use case or are there any tuning parameters which can help for eg. Akka tuning parameters? 

Thanks in advance,

Chirag