Flink support for multiple data centers

[Olga Luganska]

Hello,

Does Flink support multiple data center implementation and failover procedures in case one of the data centers goes down?

Another question I have is about data encryption. If  application state which needs to be checkpointed contains data elements which are considered to be a personally identifiable information, or maybe some credit card information, do you provide any encryption mechanisms to make sure that this data will be secured?

Thank you very much,

Olga 

[Andrey Zagrebin]

Hi Olga,

At the moment Flink does not have any embedded support for multi region deployment or failover.

You can try to automate it, for example, using savepoints [1] and backing them up in multiple data centres.

This way you could restore failed job in other regions from the latest savepoint.

Of course, job input/output should be also available in multiple data centres.

Flink does not have any embedded encryption mechanism for state.

The state value is just de-serialisable byte array for Flink.

For example, you can implement custom TypeSerializer [2] or wrap existing one with encryption.

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/savepoints.html

[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/custom_serialization.html

Best,

Andrey

On 2 Oct 2018, at 02:57, Olga Luganska <[hidden email]> wrote:

Hello,

Does Flink support multiple data center implementation and failover procedures in case one of the data centers goes down?

Another question I have is about data encryption. If  application state which needs to be checkpointed contains data elements which are considered to be a personally identifiable information, or maybe some credit card information, do you provide any encryption mechanisms to make sure that this data will be secured?

Thank you very much,

Olga