Deploying K3s HA with HAProxy Load Balancer

K3s Highly Available Architecture

Node Architecture Overview

To ensure a robust High Availability (HA) K3s cluster, the setup includes dedicated nodes for load balancing, control plane, and workloads.

🔹 Load Balancer Layer

• Purpose: Distributes traffic across control plane nodes and ensures availability.

• Nodes: 2

• Configuration:

  • Load Balancer 1 → 172.24.2.179

  • Load Balancer 2 → 172.24.2.180

  • Virtual IP (VIP) → 172.24.2.235

🔹 Control Plane Layer

• Purpose: Manages cluster state and provides redundancy via embedded etcd.

• Nodes: 3

• Configuration:

  • Control Plane 1 → 172.24.2.174

  • Control Plane 2 → 172.24.2.175

  • Control Plane 3 → 172.24.2.176

🔹 Worker Layer

• Purpose: Runs application workloads.

• Nodes: 2

• Configuration:

  • Worker Node 1 → 172.24.2.177

  • Worker Node 2 → 172.24.2.178

Before You Begin

• Ensure all nodes can communicate with each other

• Static IP addresses recommended

Step 1: Configure Load Balancer Nodes

To provide high availability for the K3s control plane, we will set up Keepalived (for VIP failover) and HAProxy (for load balancing).

1.1 Install Required Packages

Run the following on both load balancer nodes 172.24.2.179 & 172.24.2.180

sudo apt-get update & apt-get install haproxy keepalived -y

1.2 Keepalived Configuration

1 Configure Load Balancer 1

sudo vi /etc/keepalived/keepalived.conf

global_defs {
  enable_script_security
  script_user root
}

vrrp_script chk_haproxy {
    script '/usr/bin/killall -0 haproxy'
    interval 2
}

vrrp_instance haproxy-vip {
    interface ens35
    state MASTER
    priority 200
    virtual_router_id 51
    virtual_ipaddress {
        172.24.2.235/21
    }

    track_script {
        chk_haproxy
    }
}

2 Configure Load Balancer 2

sudo vi /etc/keepalived/keepalived.conf

global_defs {
  enable_script_security
  script_user root
}

vrrp_script chk_haproxy {
    script '/usr/bin/killall -0 haproxy'
    interval 2
}

vrrp_instance haproxy-vip {
    interface ens35
    state BACKUP
    priority 100
    virtual_router_id 51
    virtual_ipaddress {
        172.24.2.235/21
    }

    track_script {
        chk_haproxy
    }
}

On the secondary load balancer configure the node as a BACKUP by changing state from MASTER to BACKUP and assigning a lower priority of 100 instead of 200.

1.3 HAProxy Configuration

On both load balancer nodes:

sudo vi /etc/haproxy/haproxy.cfg

frontend k3s-frontend
    bind *:6443
    mode tcp
    option tcplog
    default_backend k3s-backend

backend k3s-backend
    mode tcp
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s
    server control-plane-1 172.24.2.174:6443 check
    server control-plane-2 172.24.2.175:6443 check
    server control-plane-3 172.24.2.176:6443 check

sudo systemctl restart haproxy keepalived
sudo systemctl enable haproxy keepalived

Step 2: K3s Master Nodes Installation

K3s master (control plane) nodes will be deployed in HA mode using the embedded etcd datastore.

2.1 Setting up 1st Control Node (172.24.2.174)

curl -sfL https://get.k3s.io | K3S_TOKEN=YOUR_SECRET sh -s - server \
    --cluster-init \
    --tls-san 172.24.2.235

The --cluster-init flag is used to initialize the embedded etcd cluster, while the --tls-san=172.24.2.235 option specifies the interface or Load Balancer VIP through which users will access the K3s cluster.

2.2 Setting Up Other 2 Control Pane Node

curl -sfL https://get.k3s.io | K3S_TOKEN=YOUR_SECRET sh -s - server \
--server https://172.24.2.174:6443 \
--tls-san 172.24.2.235

Step 3: Worker Node Installation

curl -sfL https://get.k3s.io | K3S_TOKEN=YOUR_SECRET sh -s - agent \
--server https://172.24.2.174:6443

Step 4: Verify Cluster Setup

kubectl get nodes

Step 5: Connect Cluster From Your Local Machine

5.1 Obtain the Correct kubeconfig from Control Plane Node

scp root@1172.24.2.174:/etc/rancher/k3s/k3s.yaml ~/.kube/config

5.2 Update the kubeconfig File

The copied kubeconfig likely points to 127.0.0.1:6443 (local to the node). Update it to use the VIP 172.24.2.235:6443

vi ~/.kube/config

clusters:
- cluster:
    certificate-authority-data: <base64-encoded-CA>
    server: https://172.24.2.235:6443  # Update this line
  name: default

Summary

This guide explains how to set up a highly available K3s cluster on-premises using embedded etcd and an external HAProxy load balancer with Keepalived for VIP failover. It covers load balancer configuration, control plane installation in HA mode, worker node setup, and cluster verification. Finally, it shows how to connect to the cluster from a local machine using the updated kubeconfig.