Flink and S3 AWS keys rotation

[Antonio Verardi]

Hi there,

I'm trying out to run Flink on Kubernetes and I run into a problem with the way Flink sets up AWS credentials to talk with S3 and the way we manage AWS secrets in my company.

To give permissions to Flink I am using AWS keys embedded in flink.conf, as per https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html#configure-access-credentials. The problem there is that we rotate daily our AWS keys in order to mitigate any eventual leak of keys. In order to make Flink pick up the new keys I understand I have to restart it, but that means downtime, especially for the jobs which have a large state to save.

I know that in Kubernetes land there are these two projects, https://github.com/uswitch/kiam and https://github.com/jtblin/kube2iam, that make possible to associate IAM policies to pods/containers. But they are not part of the "official" Kubernetes software, which kinda surprises me.

Did anyone run into a similar problem? If so, how did you solve it?

Cheers,

Antonio

[Kostas Kloudas-2]

Hi Antonio,

I am  cc'ing Till who may have something to say on this.

Cheers,

Kostas

On Thu, Feb 7, 2019 at 1:32 PM Antonio Verardi <[hidden email]> wrote:

Hi there,

I'm trying out to run Flink on Kubernetes and I run into a problem with the way Flink sets up AWS credentials to talk with S3 and the way we manage AWS secrets in my company.

To give permissions to Flink I am using AWS keys embedded in flink.conf, as per https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html#configure-access-credentials. The problem there is that we rotate daily our AWS keys in order to mitigate any eventual leak of keys. In order to make Flink pick up the new keys I understand I have to restart it, but that means downtime, especially for the jobs which have a large state to save.

I know that in Kubernetes land there are these two projects, https://github.com/uswitch/kiam and https://github.com/jtblin/kube2iam, that make possible to associate IAM policies to pods/containers. But they are not part of the "official" Kubernetes software, which kinda surprises me.

Did anyone run into a similar problem? If so, how did you solve it?

Cheers,

Antonio

[elmosca]

Hi,

You can give specific IAM instance roles to the instances running Flink. This way you never expose access keys anywhere. As the docs say, that is the recommended way (and not just for Flink, but for any service you want to use, never set it up with AWS credentials in config). IAM will transparently deal with the security, and you can be extremely restrictive on what policies you attach to the instance roles.

Cheers,

Bruno

On Thu, 7 Feb 2019 at 13:38, Kostas Kloudas <[hidden email]> wrote:

Hi Antonio,

I am  cc'ing Till who may have something to say on this.

Cheers,

Kostas

On Thu, Feb 7, 2019 at 1:32 PM Antonio Verardi <[hidden email]> wrote:

Hi there,

I'm trying out to run Flink on Kubernetes and I run into a problem with the way Flink sets up AWS credentials to talk with S3 and the way we manage AWS secrets in my company.

To give permissions to Flink I am using AWS keys embedded in flink.conf, as per https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html#configure-access-credentials. The problem there is that we rotate daily our AWS keys in order to mitigate any eventual leak of keys. In order to make Flink pick up the new keys I understand I have to restart it, but that means downtime, especially for the jobs which have a large state to save.

I know that in Kubernetes land there are these two projects, https://github.com/uswitch/kiam and https://github.com/jtblin/kube2iam, that make possible to associate IAM policies to pods/containers. But they are not part of the "official" Kubernetes software, which kinda surprises me.

Did anyone run into a similar problem? If so, how did you solve it?

Cheers,

Antonio

[Antonio Verardi]

Hi Bruno,

The problem with such a solution would be that those permissions will apply to any application running on the Kubernetes cluster, not only to Flink. Sharing resources with other applications is one of the cool things about Kubernetes and it would be ideal not to lose such a property. That's why I was looking into those IAM roles for pods/containers.

Cheers,

Antonio

On Thu, Feb 7, 2019 at 2:38 PM Bruno Aranda <[hidden email]> wrote:

Hi,

You can give specific IAM instance roles to the instances running Flink. This way you never expose access keys anywhere. As the docs say, that is the recommended way (and not just for Flink, but for any service you want to use, never set it up with AWS credentials in config). IAM will transparently deal with the security, and you can be extremely restrictive on what policies you attach to the instance roles.

Cheers,

Bruno

On Thu, 7 Feb 2019 at 13:38, Kostas Kloudas <[hidden email]> wrote:

Hi Antonio,

I am  cc'ing Till who may have something to say on this.

Cheers,

Kostas

On Thu, Feb 7, 2019 at 1:32 PM Antonio Verardi <[hidden email]> wrote:

Hi there,

I'm trying out to run Flink on Kubernetes and I run into a problem with the way Flink sets up AWS credentials to talk with S3 and the way we manage AWS secrets in my company.

To give permissions to Flink I am using AWS keys embedded in flink.conf, as per https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html#configure-access-credentials. The problem there is that we rotate daily our AWS keys in order to mitigate any eventual leak of keys. In order to make Flink pick up the new keys I understand I have to restart it, but that means downtime, especially for the jobs which have a large state to save.

I know that in Kubernetes land there are these two projects, https://github.com/uswitch/kiam and https://github.com/jtblin/kube2iam, that make possible to associate IAM policies to pods/containers. But they are not part of the "official" Kubernetes software, which kinda surprises me.

Did anyone run into a similar problem? If so, how did you solve it?

Cheers,

Antonio