Write your first Kubernetes Operator
Mor fallKubernetes Operator for what ?
Kubernetes operators help SRE extend kubernetes usage for custom resources. They integrate seamlessly with the kubernetes API, and can perform reconcile loop to react when resoruces change state. for more information visit kubebuilder .
Example of kubernetes API
In this example we will create a kubernetes custom controller that will perform the following actions:
Check if a custom service account exists in a kubernetes namespace
check if there is a deployment or statefulset
Compare labels (from input and existing ressources) if there is a match, scale the replicas to X (from input)
Project initialization
To initialize a project you have to run the following command with whatever domain* you whatever domain you want.
kubebuilder init --domain morbolt.dev --repo github.com/fallmor/kube-operator
The above command will create the basic boilerplate of kubebuilder project:
Config directory containing kustomisation resources and sample manifest to be deployed in the cluster.
Project file that describes the kubebuilder project.
Makefile to automate the build/run/test of our project.
Dockerfile.
Controller generated with kubebuilder
A kubernetes controller is an essential component in a kubernetes cluster, it performs different task to maintain the state of the submitted resources.
kubebuilder create api --group webapp --version v1 --kind ListRes --resource --controller
The above commands will generate new folders:
Internal contains our go code for the controller.
Api defines the specifications we will have in the Custom Resources Definition. In this folder we will have a file named listres.go in which we will define which key is required or not in our Custom Resources Definition.
type ListResSpec struct {
Resources string `json:"resources,omitempty"`
Namespace string `json:"namespace,omitempty"`
Labels map[string]string `json:"labels,omitempty"`
Replicas *int32 `json:"replicas,omitempty"`
}
// ListResStatus defines the observed state of ListRes.
type ListResStatus struct {
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// ListRes is the Schema for the listres API.
type ListRes struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec ListResSpec `json:"spec,omitempty"`
Status ListResStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
// ListResList contains a list of ListRes.
type ListResList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []ListRes `json:"items"`
}
Once we defined the structure of our ListRes resources, we can modify the file internal/controllers.go to add the logic of our operator.
In our example we will build an operator that watchs different resources (deployment, statefulset) and check if some specific labels are present. Then it will scale the replicas automatically to a specified number of replicas in ListeRes resources.
unc (r *ListResReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
l := log.FromContext(ctx)
listres := &webappv1.ListRes{}
if err := r.Get(ctx, req.NamespacedName, listres); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
l.Info("Resource", "Name", listres.Name, "Namespace", listres.Namespace)
getRes := listres.Spec.Resources
if getRes == "deploy" {
deploymentList := &appsv1.DeploymentList{}
err := r.List(ctx, deploymentList, &client.ListOptions{
Namespace: listres.Namespace,
})
if err != nil {
fmt.Printf("Failed to list deployments in namespace %s: %v\n", listres.Namespace, err)
return reconcile.Result{}, err
}
for _, deployment := range deploymentList.Items {
err := r.Get(ctx, types.NamespacedName{Namespace: listres.Namespace, Name: deployment.Name}, &deployment)
if err != nil {
println(err)
return reconcile.Result{}, err
}
if reflect.DeepEqual(listres.Spec.Labels, deployment.Labels) {
fmt.Println("good they are the same")
deployment.Spec.Replicas = listres.Spec.Replicas
fmt.Println(&deployment.Spec.Replicas)
err := r.Update(ctx, &deployment)
if err != nil {
fmt.Println("Cannot update")
}
}
}
}
if getRes == "statefulset" {
statefulsetList := &appsv1.StatefulSetList{}
err := r.List(ctx, statefulsetList, &client.ListOptions{
Namespace: listres.Namespace,
})
if err != nil {
fmt.Printf("Failed to list deployments in namespace %s: %v\n", listres.Namespace, err)
return reconcile.Result{}, err
}
for _, statefulset := range statefulsetList.Items {
err := r.Get(ctx, types.NamespacedName{Namespace: listres.Namespace, Name:statefulset.Name}, &statefulset)
if err != nil {
println(err)
return reconcile.Result{}, err
}
if reflect.DeepEqual(listres.Spec.Labels, statefulset.Labels) {
fmt.Println("good they are the same")
statefulset.Spec.Replicas = listres.Spec.Replicas
fmt.Println(&statefulset.Spec.Replicas)
err := r.Update(ctx, &statefulset)
if err != nil {
fmt.Println("Cannot update")
}
}
}
}
return ctrl.Result{}, nil
}
In this example, we list all deployment or statefulset in the Listres namespace and we perform a basic loop so that we can access the individual key of those resources. To compare two labels (go struct) we use the DeepEqual in the reflect package. Once a match is found we update the Deployments/Statefulsets.
To run the controller use this command:
make install run
Test
for testing purpose, we deploy a Kind cluster and create a ListRes ressources. In your kubernetes cluster, create a namespace named myns
apiVersion: webapp.morbolt.dev/v1
kind: ListRes
metadata:
labels:
app.kubernetes.io/name: kube-operator
app.kubernetes.io/managed-by: kustomize
name: listres-sample
namespace: myns
spec:
name: 3
resources: deploy
labels:
app: nginx-project
Deploy a sample nginx deployment with labels app: nginx-project
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: nginx-project
name: nginx-project
namespace: myns
spec:
replicas: 1
selector:
matchLabels:
app: nginx-project
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: nginx-project
spec:
containers:
- image: nginx
name: nginx
Once deployed, we can see the actions performed in the controllers log events.
2024-11-22T15:30:48+01:00 INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2024-11-22T15:30:48+01:00 INFO Starting EventSource {"controller": "listres", "controllerGroup": "webapp.morbolt.dev", "controllerKind": "ListRes", "source": "kind source: *v1.ListRes"}
2024-11-22T15:30:48+01:00 INFO Starting Controller {"controller": "listres", "controllerGroup": "webapp.morbolt.dev", "controllerKind": "ListRes"}
2024-11-22T15:30:48+01:00 INFO Starting workers {"controller": "listres", "controllerGroup": "webapp.morbolt.dev", "controllerKind": "ListRes", "worker count": 1}
2024-11-22T15:30:48+01:00 INFO Resource {"controller": "listres", "controllerGroup": "webapp.morbolt.dev", "controllerKind": "ListRes", "ListRes": {"name":"listres-sample","namespace":"myns"}, "namespace": "myns", "name": "listres-sample", "reconcileID": "0ba7e77b-389b-4c1b-932b-248f6b31e5ea", "Name": "listres-sample", "Namespace": "myns"}
good they are the same
the deployment nginx-project is scaled to 3
In the namespace myns we can see that the number of replicas is updated.
kind-my-cluster in ~/projects/kube-operator via 🐹 v1.23.3 via 10GiB/16GiB | 3GiB/4GiB with /bin/zsh took 0s
❯ kubectl get pod -n myns
NAME READY STATUS RESTARTS AGE
nginx-project-865b7c86dc-hz5gn 1/1 Running 0 4h7m
nginx-project-865b7c86dc-qnh6r 1/1 Running 0 6h16m
nginx-project-865b7c86dc-xdtq8 1/1 Running 0 4h2m
Conclusion
In this tutorial, we show you how to extend kubernetes API to perform custom tasks on ressources.
Kubebuilder permits us to build easily a kubernetes controller with small codes. In the next blog post we will explore more the usage of kubebuilder.
Subscribe to my newsletter
Read articles from Mor fall directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by