Using S3 as state backend

Posted by Brian Chhun on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Using-S3-as-state-backend-tp3996.html

Hello,

I'm trying to setup an HA cluster and I'm running into issues using S3 as the state backend. This is raised during startup:

2015-12-09T19:23:36.430724+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: java.io.IOException: No file system found with scheme s3, referenced in file URI 's3:///flink/recovery/blob'.

2015-12-09T19:23:36.430858+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:242)

2015-12-09T19:23:36.430989+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.blob.FileSystemBlobStore.<init>(FileSystemBlobStore.java:67)

2015-12-09T19:23:36.431297+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.blob.BlobServer.<init>(BlobServer.java:105)

2015-12-09T19:23:36.431435+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.jobmanager.JobManager$.createJobManagerComponents(JobManager.scala:1814)

2015-12-09T19:23:36.431569+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:1944)

2015-12-09T19:23:36.431690+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:1898)

2015-12-09T19:23:36.431810+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.jobmanager.JobManager$.startActorSystemAndJobManagerActors(JobManager.scala:1584)

2015-12-09T19:23:36.431933+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1486)

2015-12-09T19:23:36.432414+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1447)

2015-12-09T19:23:36.432649+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)

Is it possible to use S3 as the backend store or is only hdfs/mapfs supported?

Thanks,

Brian