1.13.1 jobmanager annotations by pod template does not work

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

1.13.1 jobmanager annotations by pod template does not work

ChangZhuo Chen (陳昌倬)

Hi,

We found that jobmanager annotations defined by pod template does not
work. However, annotations defined by kubernetes.jobmanager.annotations
[0]

This behavior is different from document [1].


[0] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#kubernetes-jobmanager-annotations
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template


--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 1.13.1 jobmanager annotations by pod template does not work

Yang Wang
Could you please share your pod template and the value of kubernetes.jobmanager.annotations?

Usually the annotations of pod template and flink config options will be merged. And the flink config
options has higher priority if you are specifying same name annotation.

I have verified in minikube and it could take effect as expected.

Best,
Yang

ChangZhuo Chen (陳昌倬) <[hidden email]> 于2021年6月11日周五 下午4:52写道:

Hi,

We found that jobmanager annotations defined by pod template does not
work. However, annotations defined by kubernetes.jobmanager.annotations
[0]

This behavior is different from document [1].


[0] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#kubernetes-jobmanager-annotations
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template


--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B
Reply | Threaded
Open this post in threaded view
|

Re: 1.13.1 jobmanager annotations by pod template does not work

ChangZhuo Chen (陳昌倬)
On Fri, Jun 11, 2021 at 11:19:09PM +0800, Yang Wang wrote:
> Could you please share your pod template and the value of
> kubernetes.jobmanager.annotations?
>
> Usually the annotations of pod template and flink config options will be
> merged. And the flink config
> options has higher priority if you are specifying same name annotation.
>
> I have verified in minikube and it could take effect as expected.

Hi,

There are other finding for this issue:

* For jobanager:
  * annotations, and labels in pod template do not work.
  * annotations, and labels in -Dkubernetes.jobmanager.* work.

* For taskmanager:
  * annotations, and labels in pod template work


The following is jobmanager pod template:

    apiVersion: batch/v1
    kind: Pod
    metadata:
      labels:
        app: jobmanager
        helm.sh/chart: <censored>
        app.kubernetes.io/name: <censored>
        app.kubernetes.io/instance: <censored>
        app.kubernetes.io/version: <censored>
        app.kubernetes.io/managed-by: Helm
      annotations:
        rt.prometheus.io/scrape: 'true'
        rt.prometheus.io/path: '/'
        rt.prometheus.io/port: '9249'


The following is pod created as jobmanager:


    Name:                 <censored>
    Namespace:            <censored>
    Priority:             2000000
    Priority Class Name:  medium
    Node:                 <censored>
    Start Time:           Fri, 11 Jun 2021 23:38:52 +0800
    Labels:               app=<censored>
                          component=jobmanager
                          pod-template-hash=55846fd8f7
                          type=flink-native-kubernetes
    Annotations:          <none>


--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 1.13.1 jobmanager annotations by pod template does not work

Yang Wang
The annotations, and labels in the pod template will only apply to the JobManager pod, not the JobManager deployment.

Best,
Yang

ChangZhuo Chen (陳昌倬) <[hidden email]> 于2021年6月11日周五 下午11:44写道:
On Fri, Jun 11, 2021 at 11:19:09PM +0800, Yang Wang wrote:
> Could you please share your pod template and the value of
> kubernetes.jobmanager.annotations?
>
> Usually the annotations of pod template and flink config options will be
> merged. And the flink config
> options has higher priority if you are specifying same name annotation.
>
> I have verified in minikube and it could take effect as expected.

Hi,

There are other finding for this issue:

* For jobanager:
  * annotations, and labels in pod template do not work.
  * annotations, and labels in -Dkubernetes.jobmanager.* work.

* For taskmanager:
  * annotations, and labels in pod template work


The following is jobmanager pod template:

    apiVersion: batch/v1
    kind: Pod
    metadata:
      labels:
        app: jobmanager
        helm.sh/chart: <censored>
        app.kubernetes.io/name: <censored>
        app.kubernetes.io/instance: <censored>
        app.kubernetes.io/version: <censored>
        app.kubernetes.io/managed-by: Helm
      annotations:
        rt.prometheus.io/scrape: 'true'
        rt.prometheus.io/path: '/'
        rt.prometheus.io/port: '9249'


The following is pod created as jobmanager:


    Name:                 <censored>
    Namespace:            <censored>
    Priority:             2000000
    Priority Class Name:  medium
    Node:                 <censored>
    Start Time:           Fri, 11 Jun 2021 23:38:52 +0800
    Labels:               app=<censored>
                          component=jobmanager
                          pod-template-hash=55846fd8f7
                          type=flink-native-kubernetes
    Annotations:          <none>


--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B
Reply | Threaded
Open this post in threaded view
|

Re: 1.13.1 jobmanager annotations by pod template does not work

ChangZhuo Chen (陳昌倬)
On Tue, Jun 15, 2021 at 04:22:07PM +0800, Yang Wang wrote:
> The annotations, and labels in the pod template will only apply to the
> JobManager pod, not the JobManager deployment.

Thanks for the information.

Is this behavior by design? In document, it looks like there is no
different between jobmanager and taskmanager when handling annotations
and labels [0].


[0] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template


--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 1.13.1 jobmanager annotations by pod template does not work

Yang Wang
Yes. It is the by-design behavior. Because the pod template is only applicable to the "pod", not other resources(e.g. deployment, configmap).

Currently, the JobManager pod is managed by deployment and the naked TaskManager pods are managed by Flink ResourceManager.
This is the root cause which makes the difference.

Best,
Yang

ChangZhuo Chen (陳昌倬) <[hidden email]> 于2021年6月15日周二 下午4:27写道:
On Tue, Jun 15, 2021 at 04:22:07PM +0800, Yang Wang wrote:
> The annotations, and labels in the pod template will only apply to the
> JobManager pod, not the JobManager deployment.

Thanks for the information.

Is this behavior by design? In document, it looks like there is no
different between jobmanager and taskmanager when handling annotations
and labels [0].


[0] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template


--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B
Reply | Threaded
Open this post in threaded view
|

Re: 1.13.1 jobmanager annotations by pod template does not work

ChangZhuo Chen (陳昌倬)
On Tue, Jun 15, 2021 at 04:40:00PM +0800, Yang Wang wrote:
> Yes. It is the by-design behavior. Because the pod template is only
> applicable to the "pod", not other resources(e.g. deployment, configmap).
>
> Currently, the JobManager pod is managed by deployment and the naked
> TaskManager pods are managed by Flink ResourceManager.
> This is the root cause which makes the difference.

Thanks for the clarification.


--
ChangZhuo Chen (陳昌倬) czchen@{czchen,debian}.org
http://czchen.info/
Key fingerprint = BA04 346D C2E1 FE63 C790  8793 CC65 B0CD EC27 5D5B

signature.asc (849 bytes) Download Attachment