Kubernetes CRD validation with CEL and kubebuilder marker comments

Kubernetes comes with resources like Pods, Deployments, Configmaps, PersistentVolumes & many more. Kubernetes is extensible & allows users to create Custom Resources (CR). Before creating a CR, it's required to create a Custom Resource Definition (CRD) that will define the structure & meta-attributes of the future resource.

For example, the Prometheus Operator adds a "PrometheusRule" resource. Alerts can be defined and edited with kubectl edit prometheusrule some-rule.

- Natan Yellin, CEO Robusta.dev

There are many ways to create CRDs for Kubernetes. You can write CRDs from scratch as well, but some amazing tools out there will scaffold the skeleton structure for you to make things easier. A few of them are kubebuilder, operator-sdk, etc.

Kubernetes operators require you to define & create CRDs. Often, when you develop operators, the basic requirement would be validating different fields in CRDs. There are many ways in kubebuilder to perform basic validations like setting maximum/minimum length of a field, required/optional validation checks for a field, etc.

Before Kubernetes 1.25, the only way to create complex validations in CRDs was to write & deploy a validating webhook. Each CRD would have its own validating webhook deployment running on the system. This is an operational & development overhead when you have to develop & deploy numerous CRDs. This issue is addressed in the Kubernetes 1.25 release with the introduction of CEL(Common Expression Language) validation rules. In this post, we will see the process of creating immutable CRDs before & after introduction of CEL in Kubernetes.

Warning!!!

Kindly note this feature is still in the beta phase & is subject to changes. The following sections assume you know basics of Golang, Kubernetes, & operator development.

Task

In this demo, we will try to create an immutable CRD, i.e., no one can edit the object once the CR is made. If someone tries to edit the object, the API server must reject the change & throw an error.

For this demo, we will use kubebuilder. Code available here - rewanthtammana/crd-immutable-validation-webhook

CRD validation in Kubernetes 1.23

As mentioned above, we need to create a webhook for validation. But before that, let's scaffold the skeleton.

1. Create a Kubernetes 1.23 cluster

2. Create a repository & initialize it with go mod.

    mkdir /tmp/one cd /tmp/one go mod init one
   

3. Initialize kubebuilder

   On M1,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana" --plugins=go/v4-alpha
   

   Others,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana"
   

4. Create an API with ImmutableKind. Say yes to creating a controller & resource.

    kubebuilder create api --version v1 --group validate --kind ImmutableKind
   

5. Create a webhook for validation

    kubebuilder create webhook --group validate --version v1 --kind ImmutableKind --programmatic-validation
   

6. Add the validating webhook logic to the codebase, api/v1/immutablekind_webhook.go. In this case, we want our object to be immutable, so all update operations must be blocked.

    // ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
    func (r *ImmutableKind) ValidateUpdate(old runtime.Object) error {
    immutablekindlog.Info("validate update", "name", r.Name)
   
    return apierrors.NewForbidden(
        schema.GroupResource{
            Group:    "validate.rewanthtammana.com",
            Resource: "ImmutableKind",
        }, r.Name, &field.Error{
            Type:     field.ErrorTypeForbidden,
            Field:    "*",
            BadValue: r.Name,
            Detail:   "Invalid value: \"object\": Value is immutable",
        },
    )
    }
   

7. The default webhook will not work because of certificate issues. To fix this, you must install cert-manager for operational ease.

    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml
   

8. Edit configurations from the skeleton to enable webhook, perform cainjection, etc. deployments

9. Uncomment patches/webhook_in_immutablekinds.yaml and patches/cainjection_in_immutablekinds.yaml in config/crd/kustomization.yaml

10. Uncomment ../certmanager and ../webhook directories, manager_webhook_patch.yaml & entire CERTMANAGER replacements block in config/default/kustomization.yaml

11. Create custom CRDs

    make manifests
    

12. Build & push the webhook code logic to dockerhub. In this case, I'm pushing the image to my personal dockerhub account for deployment. You can change the image name accordingly.

    make docker-build docker-push IMG=rewanthtammana/immutablekindwebhook:v1
    

13. Install & deploy the CRD & webhook

    make install deploy IMG=rewanthtammana/immutablekindwebhook:v1
    

14. Deploy a sample CR.

    kubectl apply -f ./config/samples/validate_v1_immutablekind.yaml
    

    apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
    labels:
    app.kubernetes.io/name: immutablekind
    app.kubernetes.io/instance: immutablekind-sample
    app.kubernetes.io/part-of: immutable-validation-webhook
    app.kuberentes.io/managed-by: kustomize
    app.kubernetes.io/created-by: immutable-validation-webhook
    mutate: maybe
    name: immutablekind-sample
    spec:
    # TODO(user): Add fields here
    

15. To validate the immutability feature let's edit the deployed CR.

16. To keep things simple, let's remove all labels from the above snippet & just deploy the immutablekind-sample CR again.

    echo "apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
      name: immutablekind-sample" | kubectl apply -f-
    

17. The Kubernetes API server should throw an error blocking the update.

    

CRD validation in Kubernetes 1.25

With the introduction of fantastic CEL, we can see a clear difference in the complexity of implementing CRD validation. The initial scaffolding steps are gonna remain the same.

1. Create a Kubernetes 1.25 cluster

2. Create a repository & initialize it with go mod.

    mkdir /tmp/two
    cd /tmp/two
    go mod init two
   

3. Initialize kubebuilder

   On M1,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana" --plugins=go/v4-alpha
   

   Others,

    kubebuilder init --domain rewanthtammana.com --license none --owner "rewanthtammana"
   

4. Create an API with ImmutableKind. Say yes to creating a controller & resource.

    kubebuilder create api --version v1 --group validate --kind ImmutableKind
   

5. No need to create a webhook for CRD validation with CEL

6. We aren't validating a specific field in this task. We want to protect the entire object & all its subsequent fields

7. The best way to achieve our goal is to embed the kubebuilder marker comments for the entire kind struct object

8. The CEL immutable validation check looks as below

    // +kubebuilder:validation:XValidation:rule="self == oldSelf", message="Value is immutable"
   

9. The above marker comment in CEL format is parsed by controller-gen to generate CRDs. The XValidation field in CEL rule translates to x-Kubernetes-validation in the CRD

10. The validation rule specified above, ensures that the new request object (self) is always equal to the old deployed object (oldSelf). If it's any different, the CEL validation throws an error message

11. A lot more granular validation on each field is possible with CEL. But it's not required for our demo use case

12. In this case, we created ImmutableKind struct & want to make sure it's CRs are immutable. Add the above validation marker comments to the struct

13. The ImmutableKind struct exists in api/v1/immutablekind_types.go

    // +kubebuilder:validation:XValidation:rule="self == oldSelf", message="Value is immutable"
    type ImmutableKind struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    
    Spec   ImmutableKindSpec   `json:"spec,omitempty"`
    Status ImmutableKindStatus `json:"status,omitempty"`
    }
    

14. Create custom CRDs & install them

    make manifests
    make install
    

15. Deploy a sample CR

    kubectl apply -f ./config/samples/validate_v1_immutablekind.yaml
    

    apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
    labels:
    app.kubernetes.io/name: immutablekind
    app.kubernetes.io/instance: immutablekind-sample
    app.kubernetes.io/part-of: immutable-validation-webhook
    app.kuberentes.io/managed-by: kustomize
    app.kubernetes.io/created-by: immutable-validation-webhook
    mutate: maybe
    name: immutablekind-sample
    spec:
    # TODO(user): Add fields here
    

16. To validate the immutability feature, let's edit the deployed CR.

17. To keep things simple, let's remove all labels from the above snippet & just deploy the immutablekind-sample CR again.

    echo "apiVersion: validate.rewanthtammana.com/v1
    kind: ImmutableKind
    metadata:
      name: immutablekind-sample" | kubectl apply -f-
    

18. The object update request will be failed.

    

Peek into marker comments magic

Just a one-line marker comment, removed all the complexity of creating a webhook deployment, certificate management/requirement of cert-manager deployment, etc.

The CRD configuration is located in ./config/crd/bases/validate.rewanthtammana.com_immutablekinds.yaml

The above marker comment embeds x-Kubernetes-validations field to openAPIV3Schema when you generate the manifest files.

Conclusion

This is just the tip of the iceberg. We can achieve numerous other things with the combination of CEL & Kubernetes. You can check the references for further usage.

References

• Demo codebase

• CEL

• Kubebuilder

• Kubernetes Blog