HashiCorp Vault with Docker Compose: Three-Node Cluster Setup (Part 3)

In Part 1, you set up a simple Vault instance in development mode, and in Part 2, you added persistent storage, manual unsealing, and MySQL dynamic secrets. Now, in Part 3 of our series, you’ll create a three-node Vault cluster using Docker Compose to achieve high availability (HA). A cluster ensures Vault remains accessible if a node fails, making it more production-ready. This guide walks you through setting up three Vault nodes with individual configuration files and manual joining, keeping it beginner-to-intermediate friendly.

Cover Image: A digital padlock on a circuit board, symbolizing secure secrets management. (Source: Unsplash)

Prerequisites

Before starting, ensure you have:

Docker (version 20.10 or later). Install from Docker’s guide.
Docker Compose (version 2.0 or later). See Docker Compose installation.
Vault CLI for CLI access. Download from releases.hashicorp.com/vault or use a package manager (e.g., brew install vault on macOS).
A text editor (e.g., VS Code).
Familiarity with Parts 1 and 2 (development mode, persistent storage, unsealing).
Commands are tested on Linux/macOS; Windows users may need to use set instead of export for environment variables.

Setting Up a Three-Node Vault Cluster

A Vault cluster consists of multiple nodes sharing the same storage backend, with one node acting as the leader and others as followers. If the leader fails, a follower takes over. Below is a diagram of the setup:

graph TD
    A[Client: Browser/CLI] --> B[Docker Compose]
    B --> C[Vault Node 1: Port 8200]
    B --> D[Vault Node 2: Port 8201]
    B --> E[Vault Node 3: Port 8202]
    C --> F[Shared Storage]
    D --> F
    E --> F

Step 1: Create the Docker Compose File

Create a file named docker-compose.yml in a new directory (e.g., vault-cluster):

version: '3.8'

services:
  vault1:
    image: hashicorp/vault:latest
    container_name: vault1
    ports:
      - "8200:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
      - ./vault-config/vault1:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault2:
    image: hashicorp/vault:latest
    container_name: vault2
    ports:
      - "8201:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
      - ./vault-config/vault2:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault3:
    image: hashicorp/vault:latest
    container_name: vault3
    ports:
      - "8202:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
      - ./vault-config/vault3:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

volumes:
  vault-data:

networks:
  vault-net:
    driver: bridge

Key Notes:

Three Vault services (vault1, vault2, vault3) share a vault-data volume for consistent storage.
Each node maps a unique config directory (vault-config/vault1, etc.) and exposes a different host port (8200, 8201, 8202).
vault-net: A custom network ensures nodes can communicate.
Inside containers, Vault listens on port 8200, mapped to unique host ports.

Step 2: Create Vault Configuration Files

Create a vault-config directory in vault-cluster with three subdirectories: vault1, vault2, and vault3. Add a vault.hcl file in each:

vault-config/vault1/vault.hcl:

storage "file" {
  path = "/vault/data"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

api_addr = "http://vault1:8200"
cluster_addr = "http://vault1:8201"
ui = true

vault-config/vault2/vault.hcl:

storage "file" {
  path = "/vault/data"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

api_addr = "http://vault2:8200"
cluster_addr = "http://vault2:8201"
ui = true

vault-config/vault3/vault.hcl:

storage "file" {
  path = "/vault/data"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

api_addr = "http://vault3:8200"
cluster_addr = "http://vault3:8201"
ui = true

Key Notes:

storage "file": All nodes share the vault-data volume for consistent data.
api_addr: Specifies the node’s API address for client communication.
cluster_addr: Defines the address for cluster communication (port 8201 to avoid conflicts with 8200).
tls_disable = 1: Disables TLS for simplicity; enable TLS in production.

Step 3: Start and Initialize the Cluster

Navigate to vault-cluster and start the containers:
```
 docker-compose up -d
```
Verify the containers are running:
```
 docker ps
```
You should see vault1, vault2, and vault3.

Initialize the leader node (vault1):

 export VAULT_ADDR=http://localhost:8200
 vault operator init -key-shares=1 -key-threshold=1

 Unseal Key 1: <unseal-key>
 Initial Root Token: <root-token>

 Vault initialized with 1 key shares and a key threshold of 1.
 Please securely distribute the key shares printed above.

This outputs one unseal key and a root token. Save the unseal key and root token securely (e.g., in a password manager). As in Part 2, we’re using a single key for simplicity, but production clusters typically use multiple keys (e.g., three out of five) for security. Do not use a single key in production.
Unseal vault1:
```
 vault operator unseal <unseal-key>
```

 export VAULT_TOKEN=<root-token>
 vault login

Join vault2 to the cluster:

 export VAULT_ADDR=http://localhost:8201
 vault operator raft join http://vault1:8200
 vault operator unseal <unseal-key>

Join vault3 to the cluster:

 export VAULT_ADDR=http://localhost:8202
 vault operator raft join http://vault1:8200
 vault operator unseal <unseal-key>

Step 4: Test the Cluster

Store a secret on vault1:

 export VAULT_ADDR=http://localhost:8200
 vault kv put secret/my-app db_password=supersecret

Retrieve it from vault2:

 export VAULT_ADDR=http://localhost:8201
 vault kv get secret/my-app

Check cluster status:

 vault operator raft list-peers

 Node      Address          State       Voter
 ----      -------          -----       -----
 vault1    vault1:8201      leader      true
 vault2    vault2:8201      follower    true
 vault3    vault3:8201      follower    true

This shows all three nodes, with vault1 as the leader.

Step 5: Access Vault

Web UI: Open http://localhost:8200, http://localhost:8201, or http://localhost:8202. Log in with the root token. Navigate to “Secrets” to view or create secrets.
CLI: Use any node’s address to interact with the cluster (e.g., VAULT_ADDR=http://localhost:8200).
Secrets persist across restarts due to the shared vault-data volume.

Step 6: Stop the Cluster

To stop the containers:

docker-compose down

On restart, unseal each node using the same unseal key.

Best Practices

Secure Unseal Key and Root Token: Store the unseal key and root token in a secure location (e.g., password manager or HSM). In production, use multiple keys (e.g., three out of five) per node.
Ensure Network Reliability: Nodes must communicate via cluster_addr. Use a stable network (e.g., vault-net) and monitor connectivity.
Monitor Leader Election: Check vault operator raft list-peers to ensure a leader is active. If the leader fails, a follower takes over automatically.
Enable TLS: In production, set tls_disable = 0 in vault.hcl and use certificates for secure communication.
Backup Storage: Regularly back up the vault-data volume to prevent data loss.

Troubleshooting

Node Fails to Join: Verify vault1 is unsealed and accessible (curl http://vault1:8200/v1/sys/health). Check Docker network (vault-net) with docker network inspect vault-net.
Unseal Fails: Ensure the unseal key is correct. Re-run vault operator init -key-shares=1 -key-threshold=1 on vault1 if lost (this resets the cluster).
Cluster Not Responding: Check logs (docker logs vault1) for errors. Ensure ports 8200-8202 are free (lsof -i :8200-8202).

What’s Next?

You’ve built a three-node Vault cluster with manual joining! This setup provides high availability but requires manual configuration. In Part 4, we’ll simplify clustering with retry_join for automatic node discovery. Try these experiments:

Store a secret on one node and retrieve it from another.
Stop vault1 and verify a new leader is elected (vault operator raft list-peers).

For dynamic secrets (e.g., MySQL), revisit Part 2 and adapt the configuration for this cluster. Share your progress in the comments or join the HashiCorp Community Forum!

Resources

HashiCorp Vault Documentation: vaultproject.io
Vault HA Documentation: vaultproject.io/docs/concepts/ha
Docker Compose Documentation: docs.docker.com/compose

Note: For questions or setup help, comment below or check the Vault documentation.