question related to store location of save point when using file:///, instead of s3 and hdfs

[Hung]

Hi Flink users,

When using `file:///mysavepoint` instead of `hdsf://mysavepoint  for
ENVCONFIG.setString("state.checkpoints.dir", stateBackend);

savepoint will be transfered from task manager to job manager (same as using memory as state backend), and stored in the disk of job manager?

Savepoint seems to be necessary to be stored in the place where job manager can find, and then it can start the job again. It looks like it can work without s3 or hdfs , is that true? ( if this works, that means we can use EBS)

Best,

Sendoh

[Fabian Hueske-2]

Hi,

if you use the MemoryStateBackend, the state is shipped from the TMs to the JM. The JM holds the state in its own memory (i.e., on the JVM heap).

This means that the MemoryStateBackend is not reliable by design and rather meant for testing and running Flink in your IDE.

If you want to persist the state in a distributed filesystem, you should use the FSStateBackend. Similar to the MemoryStateBackend, the FSStateBackend stores the data on the TM's heap, but on a checkpoint the state is written into a configurable FS directory.

Best, Fabian

2017-03-28 18:00 GMT+02:00 Sendoh <[hidden email]>:

Hi Flink users,

When using `file:///mysavepoint` instead of `hdsf://mysavepoint  for
ENVCONFIG.setString("state.checkpoints.dir", stateBackend);

savepoint will be transfered from task manager to job manager (same as using
memory as state backend), and stored in the disk of job manager?

Savepoint seems to be necessary to be stored in the place where job manager
can find, and then it can start the job again. It looks like it can work
without s3 or hdfs , is that true? ( if this works, that means we can use
EBS)

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/quesiont-related-to-store-location-of-save-point-when-using-file-instead-of-s3-and-hdfs-tp12435.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.