Configuration Best Practices for Batch Processing

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Configuration Best Practices for Batch Processing

Micah Whitacre
I was curious suggested best practices as it relates to running batch processes on Flink.  Does anyone have any good guides on good default settings and configuration?

One question I'm really curious about is what suggestions there might be for the relationship of memory of TaskManagers? Number of slots? It is it better to have larger TaskManagers with more slots?  Smaller TaskManagers with fewer slots?  Are there any settings related to stuff like netty/akka that are known to be good to tune to higher values for batch that wouldn't be appropriate for streaming?

Thanks,
Micah