Using Projectsveltos to Manage Kubernetes Add-ons on Civo Cloud Clusters

Civo Cloud is a cloud computing platform that provides developers with a simple and predictable way to deploy and manage their applications. Civo is focused on providing a developer-friendly experience, with a focus on Kubernetes and other cloud-native technologies.

Projectsveltos is a Kubernetes add-on controller that simplifies the deployment and management of add-ons and applications across multiple clusters. It runs in the management cluster and can programmatically deploy and manage add-ons and applications on any cluster in the fleet, including the management cluster itself. Sveltos supports a variety of add-on formats, including Helm charts, raw YAML, Kustomize, Carvel ytt, and Jsonnet.

Projectsveltos allows add-ons and applications to be represented as templates. Before deploying to managed clusters, Sveltos instantiates these templates. Sveltos can gather the information required to instantiate the templates from either the management cluster or the managed clusters themselves.

Create Kubernetes clusters on Civo cloud

Deploying Kubernetes clusters on Civo Cloud was a breeze. In my case, I only needed to specify the number of nodes and the Kubernetes version, and within a few minutes, all clusters were up and running.

Using one of the Kubernetes clusters as the management cluster, I deployed Projectsveltos. The management cluster is used by Projectsveltos to manage add-ons and applications in other Kubernetes clusters.

To deploy Projectsveltos, I pointed kubectl to the management cluster and ran the following commands:

kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/manifest.yaml
kubectl apply -f https://raw.githubusercontent.com/projectsveltos/sveltos/main/manifest/default-classifier.yaml

Once Projectsveltos was deployed, I easily registered the other Kubernetes clusters with it using sveltosctl. I still pointed kubectl to the management cluster and ran the following commands:

sveltosctl register cluster --namespace=eng --cluster=civo --kubeconfig=<civo-eng-kubeconfig>
sveltosctl register cluster --namespace=qe --cluster=civo --kubeconfig=<civo-qe-kubeconfig>
sveltosctl register cluster --namespace=marketing --cluster=civo --kubeconfig=<civo-marketing-kubeconfig>

Each of these commands created a SveltosCluster instance in the management cluster. Each SveltosCluster instance represents a Kubernetes cluster managed by Projectsveltos.

I then added labels to the clusters:

kubectl get sveltoscluster -A --show-labels
NAMESPACE   NAME   READY   VERSION        LABELS
eng         civo   true    v1.28.2+k3s1   department=eng,env=production,sveltos-agent=present
marketing   civo   true    v1.28.2+k3s1   department=marketing,env=production,sveltos-agent=present
qe          civo   true    v1.28.2+k3s1   department=qe,env=production,sveltos-agent=present

Deploy add-ons

Using Projectsveltos in the management cluster, we created ClusterProfile instances to specify the add-ons and applications to deploy to each cluster.

We created a ClusterProfile for Kyverno to deploy it to all production clusters.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: kyverno
spec:
  clusterSelector: env=production
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v3.0.3
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install
    values: |
      admissionController:
        replicas: 3

We created two ClusterProfiles, one for Prometheus and Grafana, and one for Kubevela, to deploy them to the QE cluster.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: prometheus-grafana
spec:
  clusterSelector: department=qe
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://prometheus-community.github.io/helm-charts
    repositoryName:   prometheus-community
    chartName:        prometheus-community/prometheus
    chartVersion:     23.4.0
    releaseName:      prometheus
    releaseNamespace: prometheus
    helmChartAction:  Install
  - repositoryURL:    https://grafana.github.io/helm-charts
    repositoryName:   grafana
    chartName:        grafana/grafana
    chartVersion:     6.58.9
    releaseName:      grafana
    releaseNamespace: grafana
    helmChartAction:  Install

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile 
metadata: 
  name: kubevela-core
spec:
  clusterSelector: department=qe
  syncMode: Continuous
  helmCharts:
  - repositoryURL: https://kubevela.github.io/charts
    repositoryName: kubevela
    chartName: kubevela/vela-core
    chartVersion: 1.9.6
    releaseName: kubevela-core-latest
    releaseNamespace: vela-system
    helmChartAction: Install

We created a ClusterProfile for Nginx to deploy it to the engineering cluster.

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: nginx
spec:
  clusterSelector: department=eng
  helmCharts:
  - repositoryURL:    https://helm.nginx.com/stable/
    repositoryName:   nginx-stable
    chartName:        nginx-stable/nginx-ingress
    chartVersion:     0.14.0
    releaseName:      nginx-latest
    releaseNamespace: nginx
    helmChartAction:  Install

To get a summary of what was deployed to each cluster, we used sveltosctlwith kubeconfig pointing to the management cluster:

sveltosctl show addons  
+----------------+---------------+-------------+----------------------+---------+-------------------------------+--------------------+
|    CLUSTER     | RESOURCE TYPE |  NAMESPACE  |         NAME         | VERSION |             TIME              |  CLUSTER PROFILES  |
+----------------+---------------+-------------+----------------------+---------+-------------------------------+--------------------+
| eng/civo       | helm chart    | kyverno     | kyverno-latest       | 3.0.3   | 2023-11-01 14:19:52 +0100 CET | kyverno            |
| eng/civo       | helm chart    | nginx       | nginx-latest         | 0.14.0  | 2023-11-01 14:20:39 +0100 CET | nginx              |
| qe/civo        | helm chart    | kyverno     | kyverno-latest       | 3.0.3   | 2023-11-01 14:19:46 +0100 CET | kyverno            |
| qe/civo        | helm chart    | vela-system | kubevela-core-latest | 1.9.6   | 2023-11-01 14:22:38 +0100 CET | kubevela-core      |
| qe/civo        | helm chart    | prometheus  | prometheus           | 23.4.0  | 2023-11-01 14:23:14 +0100 CET | prometheus-grafana |
| qe/civo        | helm chart    | grafana     | grafana              | 6.58.9  | 2023-11-01 14:23:17 +0100 CET | prometheus-grafana |
| marketing/civo | helm chart    | kyverno     | kyverno-latest       | 3.0.3   | 2023-11-01 14:20:01 +0100 CET | kyverno            |
+----------------+---------------+-------------+----------------------+---------+-------------------------------+--------------------+

👏 Support this project

If you enjoyed this article, please check out the Projectsveltos GitHub repo.

You can also star 🌟 the project if you found it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and examples. You can also find the latest news and updates on the project on the GitHub repo.

Thank you for reading!