S3 Input/Output with temporary credentials (IAM Roles)

[Vladimir Stoyak]

Our setup involves AWS IAM roles when with permanent access_key and access_secret we need to assume specific role (ie getting temporary credentials to use AWS resources).

I was wondering what would be the best way handling this, ie how to set fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey programmatically and also how to handle expired sessions.

Thanks,

Vladimir

[rmetzger0]

Hi Vladimir,

Flink is using Hadoop's S3 File System implementation. It seems that this feature is not supported by their implementation: https://issues.apache.org/jira/browse/HADOOP-9680

This issue contains some more information: https://issues.apache.org/jira/browse/HADOOP-9384 It seems that the s3a implementation is the one where to implement the feature.

And realistically, I don't see Hadoop fixing this in the foreseeable future :)

I see the following options:
- We could try adding the feature to Hadoop's s3a implementation. It'll probably be a few months until the fix is reviewed, merged and released (but you could probably extract the relevant code into your own project and run it from there)

- You implement an s3 file system implementation for Flink with the required features (that is not as hard as it sounds).

Sorry that I can not give you a better solution for this.

Regards,

Robert

On Fri, Dec 11, 2015 at 3:42 PM, Vladimir Stoyak <[hidden email]> wrote:

Our setup involves AWS IAM roles when with permanent access_key and access_secret we need to assume specific role (ie getting temporary credentials to use AWS resources).

I was wondering what would be the best way handling this, ie how to set fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey programmatically and also how to handle expired sessions.

Thanks,

Vladimir