Using standalone single node without HA in production, crazy?

[Ryan Crumley]

Hi,

I am evaluating flink for use in stateful streaming application. Some information about the intended use:

 - Will run in a mesos cluster and deployed via marathon in a docker container

 - Initial throughput ~ 100 messages per second (from kafka)

 - Will need to scale to 10x that soon after launch

 - State will be much larger than memory available

In order to quickly get this out the door I am considering postponing the YARN / HA setup of a cluster with the idea that the current application can easily fit within a single jvm and handle the throughput. Hopefully by the time I need more scale flink support for mesos will be available and I can use that to distribute the job to the cluster with minimal code rewrite. 

Questions:

1. Is this a viable approach? Any pitfalls to be aware of?

2. What is the correct term for this deployment mode? Single node standalone? Local?

3. Will the RocksDB state backend work in a single jvm mode?

4. When the single jvm process becomes unhealthy and is restarted by marathon will flink recover appropriately or is failure recovery a function of HA?

5. How would I migrate the RocksDB state once I move to HA mode? Is there a straight forward path?

Thanks for your time,

Ryan

[Jamie Grier]

I started to answer these questions and then realized I was making an assumption about your environment.  Do you have a reliable persistent file system such as HDFS or S3 at your disposal or do you truly mean to run on a single node?

If the you are truly thinking to run on a single node only there's no way to make this guaranteed to be reliable.  You would be open to machine and disk failures, etc.

I think the minimal reasonable production setup must use at least 3 physical nodes with the following services running:

1) HDFS or some other reliable filesystem (for persistent state storage)

2) Zookeeper for the Flink HA JobManager setup

The rest is configuration..

With regard to scaling up after your initial deployment:  right now in the latest Flink release (1.0.3) you cannot stop and restart a job with a different parallelism without losing your computed state.  What this means is that if you know you will likely scale up and you don't want to lose that state you can provision many, many slots on the TaskManagers you do run, essentially over-provisioning them, and run your job now with the max parallelism you expect to need to scale to.  This will all be much simpler to do in future Flink versions (though not in 1.1) but for now this would be a decent approach.

In Flink versions after 1.1 Flink will be able to scale parallelism up and down while preserving all of the previously computed state.

-Jamie

On Fri, Jul 1, 2016 at 6:41 AM, Ryan Crumley <[hidden email]> wrote:

Hi,

I am evaluating flink for use in stateful streaming application. Some information about the intended use:

 - Will run in a mesos cluster and deployed via marathon in a docker container

 - Initial throughput ~ 100 messages per second (from kafka)

 - Will need to scale to 10x that soon after launch

 - State will be much larger than memory available

In order to quickly get this out the door I am considering postponing the YARN / HA setup of a cluster with the idea that the current application can easily fit within a single jvm and handle the throughput. Hopefully by the time I need more scale flink support for mesos will be available and I can use that to distribute the job to the cluster with minimal code rewrite. 

Questions:

1. Is this a viable approach? Any pitfalls to be aware of?

2. What is the correct term for this deployment mode? Single node standalone? Local?

3. Will the RocksDB state backend work in a single jvm mode?

4. When the single jvm process becomes unhealthy and is restarted by marathon will flink recover appropriately or is failure recovery a function of HA?

5. How would I migrate the RocksDB state once I move to HA mode? Is there a straight forward path?

Thanks for your time,

Ryan

--

[Ryan Crumley]

Jamie,

Thank you for your insight. To answer your questions I am running on AWS and have access to S3. Further I already have Zookeeper in the mix (its used by Mesos as well as Kafka). I was hoping to avoid the complexities of an automated HA setup by running a single jvm and then migrate to HA down the road. It sounds like I can't have my cake and eat it too (yet). =) 

Ryan 

On Fri, Jul 1, 2016 at 7:22 PM Jamie Grier <[hidden email]> wrote:

I started to answer these questions and then realized I was making an assumption about your environment.  Do you have a reliable persistent file system such as HDFS or S3 at your disposal or do you truly mean to run on a single node?

If the you are truly thinking to run on a single node only there's no way to make this guaranteed to be reliable.  You would be open to machine and disk failures, etc.

I think the minimal reasonable production setup must use at least 3 physical nodes with the following services running:

1) HDFS or some other reliable filesystem (for persistent state storage)

2) Zookeeper for the Flink HA JobManager setup

The rest is configuration..

With regard to scaling up after your initial deployment:  right now in the latest Flink release (1.0.3) you cannot stop and restart a job with a different parallelism without losing your computed state.  What this means is that if you know you will likely scale up and you don't want to lose that state you can provision many, many slots on the TaskManagers you do run, essentially over-provisioning them, and run your job now with the max parallelism you expect to need to scale to.  This will all be much simpler to do in future Flink versions (though not in 1.1) but for now this would be a decent approach.

In Flink versions after 1.1 Flink will be able to scale parallelism up and down while preserving all of the previously computed state.

-Jamie

On Fri, Jul 1, 2016 at 6:41 AM, Ryan Crumley <[hidden email]> wrote:

Hi,

I am evaluating flink for use in stateful streaming application. Some information about the intended use:

 - Will run in a mesos cluster and deployed via marathon in a docker container

 - Initial throughput ~ 100 messages per second (from kafka)

 - Will need to scale to 10x that soon after launch

 - State will be much larger than memory available

In order to quickly get this out the door I am considering postponing the YARN / HA setup of a cluster with the idea that the current application can easily fit within a single jvm and handle the throughput. Hopefully by the time I need more scale flink support for mesos will be available and I can use that to distribute the job to the cluster with minimal code rewrite. 

Questions:

1. Is this a viable approach? Any pitfalls to be aware of?

2. What is the correct term for this deployment mode? Single node standalone? Local?

3. Will the RocksDB state backend work in a single jvm mode?

4. When the single jvm process becomes unhealthy and is restarted by marathon will flink recover appropriately or is failure recovery a function of HA?

5. How would I migrate the RocksDB state once I move to HA mode? Is there a straight forward path?

Thanks for your time,

Ryan

--

[Ufuk Celebi]

On Fri, Jul 1, 2016 at 3:41 PM, Ryan Crumley <[hidden email]> wrote:
> Questions:
> 1. Is this a viable approach? Any pitfalls to be aware of?

The major pitfall would be future migrations as outlined by Jamie.

> 2. What is the correct term for this deployment mode? Single node
> standalone? Local?

I would say single node standalone.

> 3. Will the RocksDB state backend work in a single jvm mode?

Yes. One JVM will execute the regular JobManager and TaskManager code.

>
> 4. When the single jvm process becomes unhealthy and is restarted by
> marathon will flink recover appropriately or is failure recovery a function
> of HA?

The checkpoint state will unfortunately get lost in this case. For
this to work you have to run with HA, which requires ZooKeeper.


> 5. How would I migrate the RocksDB state once I move to HA mode? Is there a
> straight forward path?

See Jamie's answer.

[Ryan Crumley]

Thank you Jamie and Ufuk both for such helpful answers!

I will continue to explore my options and eagerly await out of the box Mesos support.

Ryan 

On Mon, Jul 4, 2016 at 5:05 AM Ufuk Celebi <[hidden email]> wrote:

On Fri, Jul 1, 2016 at 3:41 PM, Ryan Crumley <[hidden email]> wrote:
> Questions:
> 1. Is this a viable approach? Any pitfalls to be aware of?

The major pitfall would be future migrations as outlined by Jamie.

> 2. What is the correct term for this deployment mode? Single node
> standalone? Local?

I would say single node standalone.

> 3. Will the RocksDB state backend work in a single jvm mode?

Yes. One JVM will execute the regular JobManager and TaskManager code.

>
> 4. When the single jvm process becomes unhealthy and is restarted by
> marathon will flink recover appropriately or is failure recovery a function
> of HA?

The checkpoint state will unfortunately get lost in this case. For
this to work you have to run with HA, which requires ZooKeeper.


> 5. How would I migrate the RocksDB state once I move to HA mode? Is there a
> straight forward path?

See Jamie's answer.