Velero: A Complete Backup and Recovery Solution for EKS Using Terraform

In recent years, Kubernetes has emerged as the leading platform for managing containerized applications. However, with the increase in the number of containers and complexity of applications, managing Kubernetes clusters has become more challenging. One of the major challenges is backup and recovery. Velero is an open-source tool that provides a complete backup and recovery solution for Kubernetes. In this article, we will discuss Velero in detail.

What is Velero?

Velero is a Kubernetes backup and recovery tool that was originally developed by Heptio, now part of VMware. Velero allows you to back up your entire Kubernetes cluster or specific namespaces, resources, or objects within the cluster. You can then restore the data to the same or different Kubernetes cluster. Velero is also used to migrate data between Kubernetes clusters.

How does Velero work?

Velero works by taking snapshots of your Kubernetes cluster and storing them in a cloud object storage like Amazon S3 or Google Cloud Storage. These snapshots contain all the necessary metadata and configuration information required to restore the cluster to its previous state. Velero also supports incremental backups, which means that only the changes made since the last backup will be stored.

Velero uses plugins to support various cloud providers for backup storage. The plugins help to create backup data in the specific format supported by the cloud provider. Velero also supports plugins for pre and post-backup hooks. These hooks enable the execution of scripts before and after backup to perform additional operations.

Velero’s Architecture

Velero has a client-server architecture. The Velero server runs within the Kubernetes cluster, and the Velero client is used to interact with the server. The Velero client can be run from a local machine or within the Kubernetes cluster itself.

Velero also has several other components, including a backup controller, a restore controller, a garbage collector, and a volume snapshot controller. These components work together to provide a complete backup and recovery solution for Kubernetes.

Benefits of Using Velero

1. Ease of use: Velero provides a simple and intuitive interface to backup and restore Kubernetes clusters, which makes it easy for users to use.

2. Flexibility: Velero provides flexibility in terms of backup and restore options. You can backup and restore the entire Kubernetes cluster or specific namespaces, resources, or objects within the cluster.

3. Automated backup and recovery: Velero enables automated backup and recovery, which saves time and effort.

4. Incremental backups: Velero supports incremental backups, which means that only the changes made since the last backup will be stored.

5. Cloud-native backup: Velero is designed to work in cloud-native environments and supports cloud object storage like Amazon S3 or Google Cloud Storage.

Start the journey

prerequisites

1. Access to cloud provider credentials: If you’re using a cloud-based storage provider, you’ll need to have access to the appropriate credentials for that provider. This typically includes an access key and a secret key.

2. A Kubernetes cluster: Velero is designed to work with Kubernetes clusters, so you’ll need to have a working Kubernetes cluster up and running. This can be a local cluster running on your machine, or a remote cluster running in a cloud provider such as AWS.

3. Terraform Installed: Terraform is an open-source infrastructure-as-code tool that can help you create, manage, and update infrastructure resources. You will need to have Terraform installed on your machine to use it to deploy and configure Velero.

Install Velero Client:

wget https://github.com/vmware-tanzu/velero/releases/download/v1.2.0/velero-v1.2.0-linux-amd64.tar.gz
tar -zxvf velero-v1.2.0-linux-amd64.tar.gz
sudo mv velero-v1.2.0-linux-amd64/velero /usr/local/bin/

Create S3 Bucket with attached policy:

To use Velero with AWS S3, you need to create an S3 bucket and attach a policy to it that provides Velero with the required permissions to perform backups and restores.

The policy must include permissions to create, read, and delete objects in the S3 bucket, as well as permissions to manage access to the S3 bucket. Additionally, if you want to encrypt your backups, you will need to enable server-side encryption on the S3 bucket and grant Velero permission to access the encryption keys.

resource "aws_iam_role" "velero_role" {
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

data "aws_iam_policy_document" "cluster_bucket_policy" {
  statement {
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.velero_role.arn]
    }

    actions = [
      "s3:ListBucket",
      "s3:GetObject",
      "s3:DeleteObject",
      "s3:PutObject",
      "s3:AbortMultipartUpload",
      "s3:ListMultipartUploadParts",
    ]

    resources = [
      "arn:aws:s3:::velero-backup-bucket",
      "arn:aws:s3:::velero-backup-bucket/*"
    ]
  }
}

module "s3_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "2.9.0"
  bucket        = "velero-backup-bucket
  acl           = "private"
  force_destroy = true

  attach_policy = true
  policy        = data.aws_iam_policy_document.cluster_bucket_policy.json

  attach_deny_insecure_transport_policy = false

  versioning = {
    enabled = true
  }
}

Setup Velero With IRSA:

Setting up Velero with IRSA involves creating a Kubernetes service account with the necessary IAM permissions, configuring Velero to use that service account, and ensuring that Velero has the necessary AWS credentials to access the backup storage location.

resource "aws_iam_policy" "velero_policy" {
  name        = "velero-irsa-policy"
  description = "Allow to get backup of cluster"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
        "Effect": "Allow",
        "Action": [
            "kms:GenerateDataKey",
            "kms:CreateGrant",
            "kms:Decrypt",
            "kms:ReEncryptTo",
            "kms:ReEncryptFrom",
            "ec2:DescribeVolumes",
            "ec2:DescribeSnapshots",
            "ec2:CreateTags",
            "ec2:CreateVolume",
            "ec2:CreateSnapshot",
            "ec2:DeleteSnapshot",
            "s3:GetObject",
            "s3:DeleteObject",
            "s3:PutObject",
            "s3:AbortMultipartUpload",
            "s3:ListMultipartUploadParts",
            "s3:ListBucket"
        ],
        "Resource": "*"
      }
  ]
}
EOF
}

module "eks_blueprints_kubernetes_addons" {
  depends_on              = [aws_iam_policy.velero_iam_policy]
  source                  = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.17.0"
  eks_cluster_id          = "cluster_name"
  enable_velero           = true
  velero_backup_s3_bucket = "velero-backup-bucket"
  velero_irsa_policies    = [aws_iam_policy.velero_policy.arn]
}

Schedule Backup:

The Velero schedule backup feature allows you to create a recurring backup plan for your Kubernetes resources. You can specify a schedule in Cron format, which determines the frequency of backups. For example, you can schedule a backup to occur every day at a certain time, or every week on a specific day and time.

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: velero_backup_name
  namespace: velero
spec:
  schedule: "schedule_cron_time"
  template:
    includedNamespaces:
    - 'namespaces'
    includedResources:
    - '*'
    includeClusterResources: true
    snapshotVolumes: true
    storageLocation: default
    volumeSnapshotLocations:
      - default
    ttl: 24h0m0s
status:
  phase: "Enabled"
  lastBackup:
  validationErrors:

Backup Notification:

The Velero backup notification feature integrates with Slack to provide real-time notifications for backup operations. Once a backup is complete, Velero can send a message to a Slack channel, notifying your team of the status of the backup. This can help to ensure that your team is aware of any potential issues with backups, and can take action to address them as needed.

image:
  registry: ghcr.io
  repository: kubeshop/botkube
  pullPolicy: IfNotPresent
  tag: v0.16.0


sources:
  'k8s-all-events':
    displayName: "Backup Status"
    kubernetes:
      namespaces: &k8s-events-namespaces
        include:
          - "velero"
      event:
        types:
          - all
      resources:
        - type: velero.io/v1/backups
          namespaces:
            include:
              - "velero"
          event:
            types:
              - all
          updateSetting:
            includeDiff: true
            fields:
              - status.phase
communications:
  'default-group':
    slack:
      enabled: true
      channels:
        'default':
          name: 'slack_channel_name'
          notification:
            disabled: false
          bindings:
            executors:
              - kubectl-read-only
            sources:
              - k8s-all-events
      token: 'slack_token'
      notification:
        type: short

settings:
  clusterName: eks_cluster_name

Useful Commands:

1. Create Manual Backup
   velero backup create "backup name" --include-namespaces "namespace name"
2. List all Backup
   velero backup get
3. Delete Backup
   velero backup delete "backup name"
4. Describe Backup
   velero backup describe "backup name"
5. Restore Backup
   velero restore create "restore name " --from-backup "backup name"
6. Describe Restore 
   velero restore describe "restore name"
7. Manually Schedule Backup
   velero backup schedule "schedule name" --schedule "[@Every 7d]" --ttl [24h]
8. Restore from schedule backup
   velero restore schedule --from-schedule "schedule-name"

Conclusion

Velero is an essential tool for managing Kubernetes clusters. It provides a complete backup and recovery solution for Kubernetes, which makes it easy to manage and recover from disasters. With Velero, you can automate your backup and recovery process, which saves time and effort. Velero is easy to use, flexible, and supports incremental backups, which makes it a valuable addition to any Kubernetes deployment.