Ceph configuration for checkpoints?

[Julio Biason]

Hello people,

I'm looking for the configuration options required to use Ceph as a backend for checkpoints.

So far, I can see that `state.backend.fs.checkpointdir` and `state.checkpoints.dir` should be something like `ceph:///my-checkpoint-bucket`. But where do I define server and acess and secret keys? I couldn't find anything remotely related to that in the docs...

--

[Piotr Nowojski]

Hi,

Have you tried to refer to ceph documentation? http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be:

> a drop-in replacement for the Hadoop File System (HDFS)

So I would first try to configure ceph according to their documentation and try to use Flink’s built in Hadoop connector.

Piotrek

On 12 Feb 2018, at 20:11, Julio Biason <[hidden email]> wrote:

Hello people,

I'm looking for the configuration options required to use Ceph as a backend for checkpoints.

So far, I can see that `state.backend.fs.checkpointdir` and `state.checkpoints.dir` should be something like `<a href="ceph:///my-checkpoint-bucket`" class="">ceph:///my-checkpoint-bucket`. But where do I define server and acess and secret keys? I couldn't find anything remotely related to that in the docs...

--

[Martin Pompéry]

Hello Julio,

how we deploy Ceph:

1) make sure Ceph is mounted through all the nodes that make up your Flink cluster; ensure path(s) are the same for each node

2) simply reference the Ceph mountpoint(s) via file://<your path>

Martin

On Tue, Feb 13, 2018 at 10:18 AM Piotr Nowojski <[hidden email]> wrote:

Hi,

Have you tried to refer to ceph documentation? http://docs.ceph.com/docs/jewel/cephfs/hadoop/ It claims to be:

> a drop-in replacement for the Hadoop File System (HDFS)

So I would first try to configure ceph according to their documentation and try to use Flink’s built in Hadoop connector.

Piotrek

On 12 Feb 2018, at 20:11, Julio Biason <[hidden email]> wrote:

Hello people,

I'm looking for the configuration options required to use Ceph as a backend for checkpoints.

So far, I can see that `state.backend.fs.checkpointdir` and `state.checkpoints.dir` should be something like `ceph:///my-checkpoint-bucket`. But where do I define server and acess and secret keys? I couldn't find anything remotely related to that in the docs...

--