Re: CRD compatible with native and standalone mode

Posted by gaurav kulkarni on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/CRD-compatible-with-native-and-standalone-mode-tp43066p43098.html

Thanks a lot for the response, folks! I appreciate it. I plan to use native mode in future mostly for the resource management it plans to offer. Let me go through the links provided.  

@Yang Wang "Since the CR is defined in yaml[2], native and standalone could have some dedicated fields. And you could easily parse them in your K8s operator.":  Do you mean having fields that are needed for both native and standalone mode in the CRD (probably making them optional in the CRD) and each operator type (standalone/native) using fields that are relevant for it? 

Thanks,

Gaurav

On Tuesday, April 20, 2021, 08:23:34 AM PDT, Austin Cawley-Edwards <[hidden email]> wrote:

Hi Gaurav,

I think the name "Native Kubernetes" is a bit misleading – this just means that you can use the Flink CLI/ scripts to run Flink applications on Kubernetes without using the Kubernetes APIs/ kubectl directly. What features are you looking to use in the native mode?

I think it would be difficult to use this directly inside an operator, but keeping "feature parity" with it is a good goal for your CRDs. Since CRDs are essentially just a new API, the design approach should be user/ feature-first. By feature parity, I mean taking currently supported "Native Kubernetes" functionality as the feature list for your CRDs, for example:

* Allowing Secrets to be mounted as files and environment variables [1]

* Allowing templating of the JobManager and TaskManager Pods (containers, etc.) [2]

* Easy use of built-in plugins [3]

* etc.

Other "Native Kubernetes" "features", like RBAC and logs, will come "out of the box" by defining Custom Resources.

Yang's resources are a great place to start, though I'd suggest defining your API spec within the CRD explicitly[4], which will be clearer for your users and will allow for schema validation by other tools.

Best,

Austin

[1]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-secrets

[2]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#pod-template

[3]: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#using-plugins

[4]: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema

On Tue, Apr 20, 2021 at 2:13 AM Yang Wang <[hidden email]> wrote:

I think the compatibility depends on you. For example, you could have the same

CustomResourceDefinition for standalone and native Flink applications. They could

look like this[1].

Since the CR is defined in yaml[2], native and standalone could have some dedicated fields.

And you could easily parse them in your K8s operator.

[1]. https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/crd.yaml

[2]. https://github.com/wangyang0918/flink-native-k8s-operator/blob/master/deploy/cr.yaml

Best,

Yang

gaurav kulkarni <[hidden email]> 于2021年4月20日周二 上午8:57写道：

Hi, 

I plan to create a flink K8s operator which supports standalone mode, and and switch to native mode sometime later. I was wondering what are some of the approaches to ensure that CRD is compatible with both native and standalone mode? 

Thanks