flink configuration: best practice for checkpoint storage secrets

[XU Qinghui]

Hello, folks

We are trying to use S3 for the checkpoint storage, and this involves some secrets in the configuration. We tried two approaches to configure those secrets:
- in the jvm application argument for jobmanager and taskmanager, such as -Ds3.secret-key

- in the flink-conf.yaml file for jobmanager and taskmanager

Is there a third way? What's the best practice?

Thanks a lot!

Best regards,

Qinghui

[Till Rohrmann]

Hi Qinghui,

the recommended way would be to use AWS identity and access management (IAM) [1] if possible.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#configure-access-credentials

Cheers,

Till

On Wed, Oct 7, 2020 at 12:31 PM XU Qinghui <[hidden email]> wrote:

Hello, folks

We are trying to use S3 for the checkpoint storage, and this involves some secrets in the configuration. We tried two approaches to configure those secrets:
- in the jvm application argument for jobmanager and taskmanager, such as -Ds3.secret-key

- in the flink-conf.yaml file for jobmanager and taskmanager

Is there a third way? What's the best practice?

Thanks a lot!

Best regards,

Qinghui

[XU Qinghui]

Hello Till

Thanks a lot for the reply. But it turns out the IAM is applicable only when the job is running inside AWS, which is not my case (basically we are just using the S3 API provided by other services).
By reading again the flink doc, it seems it's suggesting to use the flink-conf.yaml file, though.

Best regards,

Qinghui

Le mer. 7 oct. 2020 à 18:21, Till Rohrmann <[hidden email]> a écrit :

Hi Qinghui,

the recommended way would be to use AWS identity and access management (IAM) [1] if possible.

[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/s3.html#configure-access-credentials

Cheers,

Till

On Wed, Oct 7, 2020 at 12:31 PM XU Qinghui <[hidden email]> wrote:

Hello, folks

We are trying to use S3 for the checkpoint storage, and this involves some secrets in the configuration. We tried two approaches to configure those secrets:
- in the jvm application argument for jobmanager and taskmanager, such as -Ds3.secret-key

- in the flink-conf.yaml file for jobmanager and taskmanager

Is there a third way? What's the best practice?

Thanks a lot!

Best regards,

Qinghui