Hi all, I plan to run flink jobs as docker containers in a AWS Elastic Container Service. I will have checkpointing enabled where state is stored in a s3 bucket. Each deployment will run in a per-job mode. Are there
any non-obvious downsides to running these jobs with a local execution environment so that the deployment turns into deploying a single java application? The obvious downside is that you don’t get any horizontal scalability. That’s a given and I’d have to scale up not out in this mode. I’d like to discover if there are any other negatives with this approach. Thanks, |
Hi Joseph, thanks for reaching out to us. There shouldn't be any downsides other than the one you already mentioned as far as I know. Best, Matthias On Fri, Oct 23, 2020 at 1:27 PM Joseph Lorenzini <[hidden email]> wrote:
|
Another factor to be aware of is that there's no cluster configuration file to load (the mini-cluster doesn't know where to find flink-conf.yaml), and by default you won't have a REST API endpoint or web UI available. But you can do the configuration in the code, and/or provide configuration on the command line. And both of these points can be easily overcome with something like this: Regards,David On Thu, Oct 29, 2020 at 11:30 AM Matthias Pohl <[hidden email]> wrote:
|
Free forum by Nabble | Edit this page |