Running flink in a Local Execution Environment for Production Workloads

[Joseph Lorenzini]

Hi all,

 

I plan to run flink jobs as docker containers in a AWS Elastic Container Service. I will have checkpointing enabled where state is stored in a s3 bucket. Each deployment will run in a per-job mode.  Are there any non-obvious downsides to running these jobs with a local execution environment so that the deployment turns into deploying a single java application?

 

The obvious downside is that you don’t get any horizontal scalability. That’s a given and I’d have to scale up not out in this mode. I’d like to discover if there are any other negatives with this approach.

 

Thanks,
Joe  

Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to Internet email for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of my firm shall be understood as neither given nor endorsed by it.

[Matthias]

Hi Joseph,

thanks for reaching out to us. There shouldn't be any downsides other than the one you already mentioned as far as I know.

Best,
Matthias

On Fri, Oct 23, 2020 at 1:27 PM Joseph Lorenzini <[hidden email]> wrote:

Hi all,

 

I plan to run flink jobs as docker containers in a AWS Elastic Container Service. I will have checkpointing enabled where state is stored in a s3 bucket. Each deployment will run in a per-job mode.  Are there any non-obvious downsides to running these jobs with a local execution environment so that the deployment turns into deploying a single java application?

 

The obvious downside is that you don’t get any horizontal scalability. That’s a given and I’d have to scale up not out in this mode. I’d like to discover if there are any other negatives with this approach.

 

Thanks,
Joe  

Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to Internet email for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of my firm shall be understood as neither given nor endorsed by it.

[David Anderson-3]

Another factor to be aware of is that there's no cluster configuration file to load (the mini-cluster doesn't know where to find flink-conf.yaml), and by default you won't have a REST API endpoint or web UI available. But you can do the configuration in the code, and/or provide configuration on the command line. And both of these points can be easily overcome with something like this:

String cwd = Paths.get(".").toAbsolutePath().normalize().toString();
Configuration conf = GlobalConfiguration.loadConfiguration(cwd);
env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

Regards,

David

On Thu, Oct 29, 2020 at 11:30 AM Matthias Pohl <[hidden email]> wrote:

Hi Joseph,

thanks for reaching out to us. There shouldn't be any downsides other than the one you already mentioned as far as I know.

Best,
Matthias

On Fri, Oct 23, 2020 at 1:27 PM Joseph Lorenzini <[hidden email]> wrote:

Hi all,

 

I plan to run flink jobs as docker containers in a AWS Elastic Container Service. I will have checkpointing enabled where state is stored in a s3 bucket. Each deployment will run in a per-job mode.  Are there any non-obvious downsides to running these jobs with a local execution environment so that the deployment turns into deploying a single java application?

 

The obvious downside is that you don’t get any horizontal scalability. That’s a given and I’d have to scale up not out in this mode. I’d like to discover if there are any other negatives with this approach.

 

Thanks,
Joe  

Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to Internet email for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of my firm shall be understood as neither given nor endorsed by it.