Help the planet & downscale your workloads

In line with a previous article talking about kube-green, here is another interesting initiative to optimize your workloads in a Kubernetes cluster. In this article, I'll present kube-downscaler tool and how you can manage easily the downtime & uptime of your Kubernetes components.

Kube-downscaler has a different approach than kube-green (which is based on CronJob): it's annotation oriented, you annotated the resources you want the tool to handle. Therefore, it can be almost any kind of resource and at different levels such as namespace. There are several annotations available, I'll present some of them in this article.

Install it

To install kube-downscaler on your cluster, you have to clone locally the repository

git clone https://codeberg.org/hjacobs/kube-downscaler.git

Then deploy it to your cluster. Beware, the default configuration will shut down all resources of all namespaces (except downscaler itself & kube-system namespace) based on this cron definition: Mon-Fri 07:30-20:30 CET. You can adjust/remove it in deploy/config.yaml. When ready, apply it to your cluster

# Put kube-downscaler in a dedicated namespace
kubectl create namespace downscaler
kubectl config set-context --current --namespace=downscaler
# As it use kustomization, use -k option rather than -f
kubectl apply -k deploy
# Check everything is installed correctly
kubectl get all -n downscaler

NAME                                   READY   STATUS    RESTARTS   AGE
pod/kube-downscaler-68c68d4f56-8xstm   1/1     Running   0          67s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-downscaler   1/1     1            1           68s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/kube-downscaler-68c68d4f56   1         1         1       68s

Configure it

Now that our downscaler is installed, let's apply it to some resources. First, I deploy a replicaset of 5 Nginx servers based:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-downscaler
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-production
spec:
  rules:
  - host: demo-downscaler.yodamad.fr
    http:
      paths:
      - backend:
          service:
            name: demo-downscaler
            port:
              number: 80
        pathType: Prefix
        path: /
  tls:
    - hosts:
      - demo-downscaler.yodamad.fr
      secretName: nginx-demo-tls
---
apiVersion: v1
kind: Service
metadata:
  name: demo-downscaler
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: demo-downscaler
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-downscaler
spec:
  replicas: 5
  selector:
    matchLabels:
      app: demo-downscaler
  template:
    metadata:
      labels:
        app: demo-downscaler
    spec:
      containers:
      - image: nginxdemos/hello
        name: demo-downscaler
        ports:
        - containerPort: 80

I create this pool in a dedicated namespace

# Create namespace & use it
kubectl create namespace demo-downscaler
kubectl config set-context --current --namespace=demo-downscaler
# Deploy nginx
kubectl apply -f demo-downscaler.yaml
# Check everything is up
kubectl get pods

NAME                          READY   STATUS    RESTARTS   AGE
downscaler-6677c58c8c-2mhzl   1/1     Running   0          9m12s
downscaler-6677c58c8c-7mlgt   1/1     Running   0          9m12s
downscaler-6677c58c8c-8kgt6   1/1     Running   0          9m12s
downscaler-6677c58c8c-8wwsh   1/1     Running   0          9m12s
downscaler-6677c58c8c-d7cgb   1/1     Running   0          9m12s

Now, I annotate the deployment to shutdown for 2 hours for instance

# Annotate deployment
kubectl annotate deploy demo-downscaler 'downscaler/downtime=Mon-Fri 21:30-23:30 CET' -n demo-downscaler
# Check downscale
date
❯ Jeu 15 jui 2023 21:30:06 CEST

kubectl get pods -n demo-downscaler
❯ No resources found in demo-downscaler namespace.

kubectl logs -f kube-downscaler-68c68d4f56-8xstm -n downscaler
❯ ...
2023-06-15 21:30:05,699 INFO: Scaling down Deployment demo-downscaler/demo-downscaler from 5 to 0 replicas (uptime: always, downtime: Mon-Fri 21:30-23:30 CET)

# Check upscale
date
❯ Jeu 15 jui 2023 23:31:06 CEST

kubectl get pods -n demo-downscaler
❯ NAME                               READY   STATUS    RESTARTS   AGE
demo-downscaler-644c75db9c-4qfh8   1/1     Running   0          22s
demo-downscaler-644c75db9c-6s2hq   1/1     Running   0          22s
demo-downscaler-644c75db9c-9qsrw   1/1     Running   0          22s
demo-downscaler-644c75db9c-cqw48   1/1     Running   0          22s
demo-downscaler-644c75db9c-fwv4r   1/1     Running   0          22s

kubectl logs -f kube-downscaler-68c68d4f56-8xstm -n downscaler
❯ ...
2023-06-15 23:30:06,478 INFO: Scaling up Deployment demo-downscaler/demo-downscaler from 0 to 5 replicas (uptime: always, downtime: Mon-Fri 21:30-23:30 CET)

Here I decided to annotate my deployment directly, but I can also decide to annotate a complete namespace so that all supported resources will be shut down. Let's deploy another Nginx replicaset of 5 instances in the same namespace and annotate the namespace to downscale

# Deploy 5 nginx in another deployment
kubectl apply -f demo2-downscaler.yaml
# Remove annotation on deployment
kubectl annotate deploy demo-downscaler downscaler/downtime- -n demo-downscaler
# Annotate the namespace
kubectl annotate namespace demo-downscaler 'downscaler/downtime=Mon-Fri 23:45-23:50 CET' -n demo-downscaler

# date
date
❯ Jeu 15 jui 2023 23:45:06 CEST

kubectl logs -f kube-downscaler-68c68d4f56-8xstm -n downscaler
❯ ...
2023-06-15 23:45:01,933 INFO: Scaling down Deployment demo-downscaler/demo-downscaler from 5 to 0 replicas (uptime: always, downtime: Mon-Fri 23:45-23:50 CET)
2023-06-15 23:45:02,300 INFO: Scaling down Deployment demo-downscaler/demo2-downscaler from 5 to 0 replicas (uptime: always, downtime: Mon-Fri 23:45-23:50 CET)

kubectl get pods -n downscaler
❯ No resources found in demo-downscaler namespace.

# date
date
❯ Jeu 15 jui 2023 23:53:06 CEST

kubectl logs -f kube-downscaler-68c68d4f56-8xstm -n downscaler
❯ ...
2023-06-15 23:50:05,830 INFO: Scaling up Deployment demo-downscaler/demo-downscaler from 0 to 5 replicas (uptime: always, downtime: Mon-Fri 23:45-23:50 CET)
2023-06-15 23:50:05,845 INFO: Scaling up Deployment demo-downscaler/demo2-downscaler from 0 to 5 replicas (uptime: always, downtime: Mon-Fri 23:45-23:50 CET)

kubectl get pods -n downscaler
❯ NAME                                READY   STATUS    RESTARTS   AGE
demo-downscaler-644c75db9c-85hb4    1/1     Running   0          2m20s
demo-downscaler-644c75db9c-d6ttr    1/1     Running   0          2m20s
demo-downscaler-644c75db9c-gfwhn    1/1     Running   0          2m20s
demo-downscaler-644c75db9c-t28k7    1/1     Running   0          2m20s
demo-downscaler-644c75db9c-xmg68    1/1     Running   0          2m20s
demo2-downscaler-65567d544c-89chj   1/1     Running   0          2m20s
demo2-downscaler-65567d544c-bgtjz   1/1     Running   0          2m20s
demo2-downscaler-65567d544c-kk8h4   1/1     Running   0          2m20s
demo2-downscaler-65567d544c-swzq5   1/1     Running   0          2m20s
demo2-downscaler-65567d544c-wz25z   1/1     Running   0          2m20s

We can imagine that they are critical resources. So I can annotate this deployment with a downtime annotation to tell kube-downscaler to not stop them even if there's an annotation at the namespace level.

# Annotate namespace
kubectl annotate --overwrite namespace demo-downscaler 'downscaler/downtime=Mon-Fri 23:55-23:58 CET' -n demo-downscaler
❯ namespace/demo-downscaler annotated
# Overwrite annotation at deployment level
kubectl annotate --overwrite deploy demo-downscaler 'downscaler/downtime=never' -n demo-downscaler
❯ deployment.apps/demo-downscaler annotated

# Check downtime
date
❯ Jeu 15 jui 2023 23:55:06 CEST

kubectl get pods -n demo-downscaler
❯ NAME                               READY   STATUS
demo-downscaler-644c75db9c-4rcf7   1/1     Running
demo-downscaler-644c75db9c-h8jpw   1/1     Running
demo-downscaler-644c75db9c-hbjhg   1/1     Running
demo-downscaler-644c75db9c-p2gk9   1/1     Running
demo-downscaler-644c75db9c-w2v6d   1/1     Running

kubectl logs -f kube-downscaler-68c68d4f56-8xstm -n downscaler
❯ ...
2023-06-15 20:13:07,289 INFO: Scaling up Deployment demo-downscaler/demo2-downscaler from 0 to 5 replicas (uptime: always, downtime: Mon-Fri 23:55-23:58 CET)

# Check downtime
date
❯ Jeu 15 jui 2023 23:59:06 CEST

kubectl logs -f kube-downscaler-68c68d4f56-8xstm -n downscaler
❯ ...
2023-06-15 23:58:07,289 INFO: Scaling up Deployment demo-downscaler/demo2-downscaler from 0 to 5 replicas (uptime: always, downtime: Mon-Fri 23:55-23:58 CET)

kubectl get pods -n demo-downscaler
❯ NAME                                READY   STATUS
demo-downscaler-644c75db9c-4rcf7    1/1     Running
demo-downscaler-644c75db9c-h8jpw    1/1     Running
demo-downscaler-644c75db9c-hbjhg    1/1     Running
demo-downscaler-644c75db9c-p2gk9    1/1     Running
demo-downscaler-644c75db9c-w2v6d    1/1     Running
demo2-downscaler-65567d544c-ccnhs   1/1     Running
demo2-downscaler-65567d544c-hgrqs   1/1     Running
demo2-downscaler-65567d544c-jnw5q   1/1     Running
demo2-downscaler-65567d544c-k6tvf   1/1     Running
demo2-downscaler-65567d544c-wnjgf   1/1     Running

Finally, as a last use-case, we can set up a minimum of replicas even if there is a downtime period on the deployment to ensure that there is at least always a minimum of pods up.

# Remove all annotations
kubectl annotate --overwrite deploy demo-downscaler downscaler/downtime- -n demo-downscaler
kubectl annotate --overwrite namespace demo-downscaler downscaler/downtime- -n demo-downscaler

# Annotate deployment with downtime & replicas
kubectl annotate deploy demo-downscaler 'downscaler/downtime=Mon-Fri 00:05-00:10 CET' -n demo-downscaler
kubectl annotate deploy demo-downscaler 'downscaler/downtime-replicas=2' -n demo-downscaler

# Check downtime
date
❯ Ven 16 jui 2023 00:06:06 CEST

kubectl logs -f kube-downscaler-68c68d4f56-8xstm -n downscaler
❯ ...
2023-06-16 00:05:11,394 INFO: Scaling down Deployment demo-downscaler/demo-downscaler from 5 to 2 replicas (uptime: always, downtime: Mon-Fri 00:05-00:10 CET)

kubectl get pods -n demo-downscaler
❯ NAME                                READY   STATUS
demo-downscaler-644c75db9c-4rcf7   1/1     Running
demo-downscaler-644c75db9c-p2gk9   1/1     Running

Conclusion

In conclusion, kube-downscaler brings a lot of options and flexibility to optimize your workload. Compared to kube-green, it seems to cover more use cases. But we can see a few limitations: for now, it's not CNCF certified. It is not a big deal, but this means it doesn't have the same visibility as kube-green, thus maybe it would improve less fast than certified tools. Another one is that it is based on annotation (which can be seen also as an advantage), so we cannot have several one on the same component, for instance, to handle complex periods. But it's not blocking, as we can play with cron definition which is deeply configurable. Definitely a great tool to add to our k8s toolbox in my opinion. There are other options available, but the documentation is quite well-written & complete.

ℹ️ A last precision if you want to try the tool, there is a grace period of 15 minutes by default. So resources that were created less than 15 min ago regarding the downtime period specified won't be stopped (you can easily see it in the kube-downscaler pod logs). You can overwrite this parameter when deploying the tool.

At the time of this article, kube-downscaler was in version 23.2.0. Thanks again to OVHcloud for their support in providing me with resources to try all these nice tools.