Native Kubernetes annotation parsing problem

[Kevin Kwon]

Hi team, I'm using Native Kubernetes annotation config

kubernetes.jobmanager.annotations

and I'm facing some problem with parsing.

I use annotation

iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'

but seems no matter what I do, the colon is getting parsed for key, value. can anyone help?

[Kevin Kwon]

I think it will be more generic question of how I inject IAM roles in Native Kubernetes pods

I'm using Kubeiam and seems the namespace annotation doesn't work

On Mon, Feb 8, 2021 at 2:30 PM Kevin Kwon <[hidden email]> wrote:

Hi team, I'm using Native Kubernetes annotation config

kubernetes.jobmanager.annotations

and I'm facing some problem with parsing.

I use annotation

iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'

but seems no matter what I do, the colon is getting parsed for key, value. can anyone help?

[Yang Wang]

If you are setting the config options in flink-conf.yaml, then you could directly add the following example.

kubernetes.jobmanager.annotations: iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'

However, if you are using the dynamic properties in the CLI. Then the configuration value should also

be wrapped with a double quote. It just because we need to escape the value with single quote.

-Dkubernetes.jobmanager.annotations="iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'"

It seems that IAM is not a common feature in Kubernetes. But from the documentation of AWS[1],

I think it could be specified via service account[2]. Hope this helps.

[1]. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/native_kubernetes.html#rbac

Best,

Yang

Kevin Kwon <[hidden email]> 于2021年2月8日周一 下午11:58写道：

I think it will be more generic question of how I inject IAM roles in Native Kubernetes pods

I'm using Kubeiam and seems the namespace annotation doesn't work

On Mon, Feb 8, 2021 at 2:30 PM Kevin Kwon <[hidden email]> wrote:

Hi team, I'm using Native Kubernetes annotation config

kubernetes.jobmanager.annotations

and I'm facing some problem with parsing.

I use annotation

iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'

but seems no matter what I do, the colon is getting parsed for key, value. can anyone help?

[Yang Wang]

> is there a way I can load my own ConfigMap and specify it via the dynamic argument?

Flink client will automatically ship the "flink-conf.yaml" and "log4j-console.properties" under FLINK_HOME/conf directory on the client side.

After the application is submitted successfully, you could find a ConfigMap named "flink-config-<cluster-id>".

If you want to mount your own ConfigMaps, it could not be supported now. We will get it done via pod template[1]. It is still in progress and

have a PR[2] waiting for review.

[1]. https://issues.apache.org/jira/browse/FLINK-15656

[2]. https://github.com/apache/flink/pull/14629

Best,

Yang

Kevin Kwon <[hidden email]> 于2021年2月9日周二 下午8:55写道：

thank you Yang

but is there a way I can load my own ConfigMap and specify it via the dynamic argument?

it seems that kubernetes.hadoop.conf.config-map.name exists but it's only for custom hadoop config

I'd want to encapsulate all the my custom config (flink-conf.yaml, log4j-consle.yaml etc) and feed it into native K8S CLI

On Tue, Feb 9, 2021 at 3:54 AM Yang Wang <[hidden email]> wrote:

If you are setting the config options in flink-conf.yaml, then you could directly add the following example.

kubernetes.jobmanager.annotations: iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'

However, if you are using the dynamic properties in the CLI. Then the configuration value should also

be wrapped with a double quote. It just because we need to escape the value with single quote.

-Dkubernetes.jobmanager.annotations="iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'"

It seems that IAM is not a common feature in Kubernetes. But from the documentation of AWS[1],

I think it could be specified via service account[2]. Hope this helps.

[1]. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

[2]. https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/native_kubernetes.html#rbac

Best,

Yang

Kevin Kwon <[hidden email]> 于2021年2月8日周一 下午11:58写道：

I think it will be more generic question of how I inject IAM roles in Native Kubernetes pods

I'm using Kubeiam and seems the namespace annotation doesn't work

On Mon, Feb 8, 2021 at 2:30 PM Kevin Kwon <[hidden email]> wrote:

Hi team, I'm using Native Kubernetes annotation config

kubernetes.jobmanager.annotations

and I'm facing some problem with parsing.

I use annotation

iam.amazonaws.com/role:'arn:aws:iam::XXXXXXXX:role/XX/XXXX'

but seems no matter what I do, the colon is getting parsed for key, value. can anyone help?