HashiCorp Vault with Docker Compose: Three-Node Cluster Setup (Part 3)

In Part 1, you set up a simple Vault instance in development mode, and in Part 2, you added persistent storage, manual unsealing, and MySQL dynamic secrets. While functional, a single node lacks high availability (HA). Now, in Part 3 of our series, you’ll create a three-node Vault cluster using Docker Compose to achieve high availability (HA). A cluster ensures Vault remains accessible if a node fails, making it more production-ready. This guide walks you through setting up three Vault nodes with individual configuration files and manual joining, keeping it beginner-to-intermediate friendly.
Cover Image: A digital padlock on a circuit board, symbolizing secure secrets management. (Source: Unsplash)
Prerequisites
Before starting, ensure you have:
Docker (version 20.10 or later). Install from Docker’s guide.
Docker Compose (version 2.0 or later). See Docker Compose installation.
Vault CLI for CLI access. Download from releases.hashicorp.com/vault or use a package manager (e.g.,
brew install vault
on macOS).A text editor (e.g., VS Code).
Familiarity with Parts 1 and 2 (development mode, persistent storage, unsealing).
Commands are tested on Linux/macOS; Windows users may need to use
set
instead ofexport
for environment variables.
Setting Up a Three-Node Vault Cluster
A Vault cluster consists of multiple nodes sharing the same storage backend, with one node acting as the leader and others as followers. If the leader fails, a follower takes over. Below is a diagram of the setup:
graph TD
A[Client: Browser/CLI] --> B[Docker Compose]
B --> C[Vault Node 1: Port 8200]
B --> D[Vault Node 2: Port 8201]
B --> E[Vault Node 3: Port 8202]
C --> F[Shared Storage]
D --> F
E --> F
Step 1: Create the Docker Compose File
Create a file named docker-compose.yml
in a new directory (e.g., vault-cluster
):
services:
vault1:
image: hashicorp/vault:latest
container_name: vault1
ports:
- "8200:8200"
environment:
- VAULT_ADDR=http://0.0.0.0:8200
cap_add:
- IPC_LOCK
volumes:
- vault1-data:/vault/file
- ./vault-config/vault1:/vault/config
command: vault server -config=/vault/config/vault.hcl
networks:
- vault-net
vault2:
image: hashicorp/vault:latest
container_name: vault2
ports:
- "8201:8200"
environment:
- VAULT_ADDR=http://0.0.0.0:8200
cap_add:
- IPC_LOCK
volumes:
- vault2-data:/vault/file
- ./vault-config/vault2:/vault/config
command: vault server -config=/vault/config/vault.hcl
networks:
- vault-net
vault3:
image: hashicorp/vault:latest
container_name: vault3
ports:
- "8202:8200"
environment:
- VAULT_ADDR=http://0.0.0.0:8200
cap_add:
- IPC_LOCK
volumes:
- vault3-data:/vault/file
- ./vault-config/vault3:/vault/config
command: vault server -config=/vault/config/vault.hcl
networks:
- vault-net
volumes:
vault1-data:
vault2-data:
vault3-data:
networks:
vault-net:
driver: bridge
Key Notes:
Separate volumes (vault1-data, vault2-data, vault3-data) ensure Raft storage is independent per node.
Each node maps a unique config directory (
vault-config/vault1
, etc.) and exposes a different host port (8200
,8201
,8202
).vault-net
: A custom network ensures nodes can communicate.Inside containers, Vault listens on port
8200
, mapped to unique host ports.
Step 2: Create Vault Configuration Files
To simplify cluster setup, we could manually join nodes using vault operator raft join
, but this approach is tedious and error-prone, requiring careful coordination for each node. Instead, we’ll use retry_join to automate node discovery, making the process faster, more reliable, and less boring. This configuration allows each node to automatically find and join the cluster by contacting other nodes, streamlining setup and improving resilience.
Moving to Raft storage
In Part 2, we used file storage with a shared volume across containers for simplicity, enabling a single-node Vault to persist data easily. In Part 3, we switch to Raft storage, which is integrated into Vault and designed for high availability clustering. Raft ensures each node maintains its own consistent state and supports leader election and replication, making it ideal for a resilient, multi-node cluster. Additionally, unlike Part 2’s shared volume, Part 3 uses separate storage volumes for each node (vault1-data, vault2-data, vault3-data), eliminating shared storage dependencies and aligning with Raft’s requirement for independent node data.
Create a vault-config directory with subdirectories: vault1
, vault2
, vault3
. Add vault.hcl
files following the specified pattern:
vault-config/vault1/vault.hcl:
ui = true
api_addr = "http://vault1:8200"
cluster_addr = "http://vault1:8201"
storage "raft" {
path = "/vault/file"
node_id = "vault1"
retry_join {
leader_api_addr = "http://vault2:8200"
leader_api_addr = "http://vault3:8200"
}
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
The retry_join configuration automates cluster formation by enabling each Vault node to attempt joining other nodes’ API addresses (e.g., vault1
tries vault2
and vault3
) until successful. This resilient mechanism retries on failure and excludes the current node to avoid self-joining, ensuring robust clustering. By specifying multiple leader_api_addr
entries, retry_join
simplifies setup and enhances reliability compared to manual joining.
vault-config/vault2/vault.hcl:
ui = true
api_addr = "http://vault2:8200"
cluster_addr = "http://vault2:8201"
storage "raft" {
path = "/vault/file"
node_id = "vault2"
retry_join {
leader_api_addr = "http://vault1:8200"
leader_api_addr = "http://vault3:8200"
}
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
vault-config/vault3/vault.hcl:
ui = true
api_addr = "http://vault3:8200"
cluster_addr = "http://vault3:8201"
storage "raft" {
path = "/vault/file"
node_id = "vault3"
retry_join {
leader_api_addr = "http://vault1:8200"
leader_api_addr = "http://vault2:8200"
}
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
Key Notes:
storage "raft"
: Uses Raft for HA, with a unique node_id per node.retry_join
: Excludes the current node (e.g., vault1 joins vault2 and vault3), improving resilience.tls_disable = 1
: Disabled for simplicity; production requires TLS.Production Warning: Use multiple unseal keys and TLS in production.
api_addr
: Specifies the node’s API address for client communication.cluster_addr
: Defines the address for cluster communication (port8201
to avoid conflicts with8200
).
Step 3: Start and Initialize the Cluster
Navigate to
vault-cluster
and start the containers:docker-compose up -d
Verify the containers are running:
docker ps
You should see
vault1
,vault2
, andvault3
.Initialize the leader node (
vault1
):export VAULT_ADDR=http://localhost:8200 vault operator init -key-shares=1 -key-threshold=1 Unseal Key 1: <unseal-key> Initial Root Token: <root-token> Vault initialized with 1 key shares and a key threshold of 1. Please securely distribute the key shares printed above.
This outputs one unseal key and a root token. Save the unseal key and root token securely (e.g., in a password manager). As in Part 2, we’re using a single key for simplicity, but production clusters typically use multiple keys (e.g., three out of five) for security. Do not use a single key in production.
Unseal
vault1
:vault operator unseal <unseal-key>
Log in with the root token:
export VAULT_TOKEN=<root-token> vault login
Join
vault2
to the cluster:export VAULT_ADDR=http://localhost:8201 vault operator init -key-shares=1 -key-threshold=1 vault operator unseal <vault2-unseal-key>
Join
vault3
to the cluster:export VAULT_ADDR=http://localhost:8202 vault operator init -key-shares=1 -key-threshold=1 vault operator unseal <vault3-unseal-key>
Step 4: Test the Cluster
Store a secret on
vault1
:export VAULT_ADDR=http://localhost:8200 vault kv put secret/my-app db_password=supersecret
Retrieve it from
vault2
:export VAULT_ADDR=http://localhost:8201 vault kv get secret/my-app
Check cluster status:
vault operator raft list-peers Node Address State Voter ---- ------- ----- ----- vault1 vault1:8201 leader true vault2 vault2:8201 follower true vault3 vault3:8201 follower true
This shows all three nodes, with
vault1
as the leader.
Step 5: Access Vault
Web UI: Open
http://localhost:8200
,http://localhost:8201
, orhttp://localhost:8202
. Log in with the root token. Navigate to “Secrets” to view or create secrets.CLI: Use any node’s address to interact with the cluster (e.g.,
VAULT_ADDR=http://localhost:8200
).Secrets persist via separate Raft storage volumes.
Step 6: Stop the Cluster
To stop the containers:
docker-compose down
On restart, unseal each node using the same unseal key.
Best Practices
Use Multiple Keys in Production: Configure multiple unseal keys (e.g., three out of five) for security. Avoid a single key.
Monitor Retry Join: Ensure retry_join targets are stable. Use cloud metadata or multiple addresses in production.
Backup Raft Storage: Regularly back up vault1-data, vault2-data, and vault3-data volumes.
Secure Unseal Key and Root Token: Store the unseal key and root token in a secure location (e.g., password manager or HSM). In production, use multiple keys (e.g., three out of five) per node.
Ensure Network Reliability: Nodes must communicate via
cluster_addr
. Use a stable network (e.g.,vault-net
) and monitor connectivity.Monitor Leader Election: Check
vault operator raft list-peers
to ensure a leader is active. If the leader fails, a follower takes over automatically.Enable TLS: In production, set
tls_disable = 0
invault.hcl
and use certificates for secure communication.Backup Storage: Regularly back up the
vault-data
volume to prevent data loss.
Troubleshooting
Node Fails to Join: Verify
vault1
is unsealed and accessible (curl http://vault1:8200/v1/sys/health
). Check Docker network (vault-net
) withdocker network inspect vault-net
.Raft Issues: Ensure separate volumes are mounted (docker inspect vault1). Check logs (docker logs vault1).
Unseal Fails: Ensure the unseal key is correct. Re-run
vault operator init -key-shares=1 -key-threshold=1
onvault1
if lost (this resets the cluster).Cluster Not Responding: Check logs (
docker logs vault1
) for errors. Ensure ports8200-8202
are free (lsof -i :8200-8202
).
What’s Next?
You’ve built a high-availability Vault cluster with Raft and retry_join! In Part 4, we’ll eliminate manual unsealing by implementing transit-based auto-unseal with a dedicated Vault instance. Try:
Stop
vault1
and verify a new leader is elected (vault operator raft list-peers
).Adapting MySQL secrets from Part 2.
Store a secret on one node and retrieve it from another.
For dynamic secrets (e.g., MySQL), revisit Part 2 and adapt the configuration for this cluster. Share your progress in the comments or join the HashiCorp Community Forum!
Resources
HashiCorp Vault Documentation: vaultproject.io
Vault HA Documentation: vaultproject.io/docs/concepts/ha
Raft Storage Docs: vaultproject.io/docs/configuration/storage/raft
Docker Compose Documentation: docs.docker.com/compose
Note: For questions or setup help, comment below or check the Vault documentation.
Subscribe to my newsletter
Read articles from Miroslav Milak directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
Miroslav Milak
Miroslav Milak
HashiCorp Vault SME Certified | Passionate about secure secrets management and cloud infrastructure. Sharing insights, tutorials, and best practices on HashiNode to help engineers build resilient, scalable systems. Advocate for DevSecOps and cutting-edge security solutions.