rocksdb without checkpointing

[abhishekrs]

Is it possible to set state backend as RocksDB without asking it to checkpoint?

We are trying to do application level checkpointing (since it gives us better flexibility to upgrade our flink pipeline and also restore state in a application specific upgrade friendly way). So we don’t really need rocksDB to do any checkpointing. Moreover, we also observed that there is 20s stall every hour that seems to correlate with rocksDB wanting to checkpoint.

Will the following work (effectively disable checkpointing)? 

new RocksDBStateBackend("file:///dev/null") 

Or is there a better way?

-Abhishek-

[Vinay Patil]

Hi Abhishek,

You can disable checkpointing by not commenting env.enableCheckpointing

What do you mean by "We are trying to do application level checkpointing"

On Thu, Feb 16, 2017 at 12:42 AM, abhishekrs [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Is it possible to set state backend as RocksDB without asking it to checkpoint?

We are trying to do application level checkpointing (since it gives us better flexibility to upgrade our flink pipeline and also restore state in a application specific upgrade friendly way). So we don’t really need rocksDB to do any checkpointing. Moreover, we also observed that there is 20s stall every hour that seems to correlate with rocksDB wanting to checkpoint.

Will the following work (effectively disable checkpointing)? 

new RocksDBStateBackend("file:///dev/null") 

Or is there a better way?

-Abhishek-

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/rocksdb-without-checkpointing-tp11645.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

[abhishekrs]

Sorry, that was a red herring. Checkpointing was not getting triggered because we never enabled it. 

Our application is inherently restartable because we can use our own output to rebuild state. All that is working fine for us - including restart semantics - without having to worry about upgrading flink topology. Once we have something in production, will be happy to share more details in flink forums. We are very pleased with flink so far. Some paradigms are messy (scale of select for e.g), but we are very pleased overall !!

On Wed, Feb 15, 2017 at 7:54 PM vinay patil <[hidden email]> wrote:

Hi Abhishek,

You can disable checkpointing by not commenting env.enableCheckpointing

What do you mean by "We are trying to do application level checkpointing"

On Thu, Feb 16, 2017 at 12:42 AM, abhishekrs [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Is it possible to set state backend as RocksDB without asking it to checkpoint?

We are trying to do application level checkpointing (since it gives us better flexibility to upgrade our flink pipeline and also restore state in a application specific upgrade friendly way). So we don’t really need rocksDB to do any checkpointing. Moreover, we also observed that there is 20s stall every hour that seems to correlate with rocksDB wanting to checkpoint.

Will the following work (effectively disable checkpointing)? 

new RocksDBStateBackend("file:///dev/null") 

Or is there a better way?

-Abhishek-

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/rocksdb-without-checkpointing-tp11645.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

---

View this message in context: Re: rocksdb without checkpointing
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

[Vinay Patil]

Good to hear that.

On which machine you are running your Flink Job, also what are the configurations you have used for RocksDB

I am currently running on C3.4xlarge with predefined option set to FLASH_SSD_OPTIMIZED

On Thu, Feb 16, 2017 at 10:31 AM, abhishekrs [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Sorry, that was a red herring. Checkpointing was not getting triggered because we never enabled it. 

Our application is inherently restartable because we can use our own output to rebuild state. All that is working fine for us - including restart semantics - without having to worry about upgrading flink topology. Once we have something in production, will be happy to share more details in flink forums. We are very pleased with flink so far. Some paradigms are messy (scale of select for e.g), but we are very pleased overall !!

On Wed, Feb 15, 2017 at 7:54 PM vinay patil <[hidden email]> wrote:

Hi Abhishek,

You can disable checkpointing by not commenting env.enableCheckpointing

What do you mean by "We are trying to do application level checkpointing"

On Thu, Feb 16, 2017 at 12:42 AM, abhishekrs [via Apache Flink User Mailing List archive.] <[hidden email]> wrote:

Is it possible to set state backend as RocksDB without asking it to checkpoint?

We are trying to do application level checkpointing (since it gives us better flexibility to upgrade our flink pipeline and also restore state in a application specific upgrade friendly way). So we don’t really need rocksDB to do any checkpointing. Moreover, we also observed that there is 20s stall every hour that seems to correlate with rocksDB wanting to checkpoint.

Will the following work (effectively disable checkpointing)? 

new RocksDBStateBackend("file:///dev/null") 

Or is there a better way?

-Abhishek-

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/rocksdb-without-checkpointing-tp11645.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML

---

View this message in context: Re: rocksdb without checkpointing
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

---

If you reply to this email, your message will be added to the discussion below:

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/rocksdb-without-checkpointing-tp11645p11653.html

To start a new topic under Apache Flink User Mailing List archive., email [hidden email]
To unsubscribe from Apache Flink User Mailing List archive., click here.
NAML