Rocksdb in production

[Jayant Ameta]

[Fabian Hueske-2]

Hi,
RockDB is an embedded key-value storage that is internally used by Flink. There is no need to setup a RocksDB database or service yourself. All of that is done by Flink.

As a Flink user that uses the RockDB state backend, you won't get in touch with RocksDB itself.

Besides that, RocksDB is developed by Facebook [1] and is a fairly active project.

Best, Fabian

[1] https://github.com/facebook/rocksdb

2018-03-05 3:57 GMT-08:00 Jayant Ameta <[hidden email]>:

[Jayant Ameta]

Oh! Somehow I missed while reading the documentation that RocksDB is embedded in Flink.

Also, irrespective of state backend being filesystem or rocksdb, I'll have to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct?

On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <[hidden email]> wrote:

Hi,
RockDB is an embedded key-value storage that is internally used by Flink. There is no need to setup a RocksDB database or service yourself. All of that is done by Flink.

As a Flink user that uses the RockDB state backend, you won't get in touch with RocksDB itself.

Besides that, RocksDB is developed by Facebook [1] and is a fairly active project.

Best, Fabian

[1] https://github.com/facebook/rocksdb

2018-03-05 3:57 GMT-08:00 Jayant Ameta <[hidden email]>:

[Fabian Hueske-2]

Yes, that is correct.

2018-03-05 8:57 GMT-08:00 Jayant Ameta <[hidden email]>:

Oh! Somehow I missed while reading the documentation that RocksDB is embedded in Flink.

Also, irrespective of state backend being filesystem or rocksdb, I'll have to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct?

On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <[hidden email]> wrote:

Hi,
RockDB is an embedded key-value storage that is internally used by Flink. There is no need to setup a RocksDB database or service yourself. All of that is done by Flink.

As a Flink user that uses the RockDB state backend, you won't get in touch with RocksDB itself.

Besides that, RocksDB is developed by Facebook [1] and is a fairly active project.

Best, Fabian

[1] https://github.com/facebook/rocksdb

2018-03-05 3:57 GMT-08:00 Jayant Ameta <[hidden email]>:

[Jayant Ameta]

Thanks Fabian.

Will there be any performance issues if I use NFS as the shared filesystem (as compared to HDFS or S3)?

On Mon, Mar 5, 2018 at 10:31 PM, Fabian Hueske <[hidden email]> wrote:

Yes, that is correct.

2018-03-05 8:57 GMT-08:00 Jayant Ameta <[hidden email]>:

Oh! Somehow I missed while reading the documentation that RocksDB is embedded in Flink.

Also, irrespective of state backend being filesystem or rocksdb, I'll have to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct?

On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <[hidden email]> wrote:

Hi,
RockDB is an embedded key-value storage that is internally used by Flink. There is no need to setup a RocksDB database or service yourself. All of that is done by Flink.

As a Flink user that uses the RockDB state backend, you won't get in touch with RocksDB itself.

Besides that, RocksDB is developed by Facebook [1] and is a fairly active project.

Best, Fabian

[1] https://github.com/facebook/rocksdb

2018-03-05 3:57 GMT-08:00 Jayant Ameta <[hidden email]>:

[Fabian Hueske-2]

That depends on your job and the setup.
Remember that all operators will write their checkpoint data into that file system.
If the state grows very large and only have an NFS with little write performance, it might be a problem. But the same would apply to HDFS as well.

2018-03-06 2:51 GMT-08:00 Jayant Ameta <[hidden email]>:

Thanks Fabian.

Will there be any performance issues if I use NFS as the shared filesystem (as compared to HDFS or S3)?

On Mon, Mar 5, 2018 at 10:31 PM, Fabian Hueske <[hidden email]> wrote:

Yes, that is correct.

2018-03-05 8:57 GMT-08:00 Jayant Ameta <[hidden email]>:

Oh! Somehow I missed while reading the documentation that RocksDB is embedded in Flink.

Also, irrespective of state backend being filesystem or rocksdb, I'll have to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct?

On Mon, Mar 5, 2018 at 9:51 PM, Fabian Hueske <[hidden email]> wrote:

Hi,
RockDB is an embedded key-value storage that is internally used by Flink. There is no need to setup a RocksDB database or service yourself. All of that is done by Flink.

As a Flink user that uses the RockDB state backend, you won't get in touch with RocksDB itself.

Besides that, RocksDB is developed by Facebook [1] and is a fairly active project.

Best, Fabian

[1] https://github.com/facebook/rocksdb

2018-03-05 3:57 GMT-08:00 Jayant Ameta <[hidden email]>: