HashiCorp Vault with Docker Compose: Advanced Auto-Unseal with Transit (Part 4)

Subtitle: Transition from manual unsealing to auto-unseal using a dedicated transit Vault

In Part 3, you built a three-node Vault cluster using Raft storage and retry_join for automated clustering, with separate storage volumes for high availability (HA). However, each node required manual unsealing, which is cumbersome. In this final, comprehensive part of our series, we’ll enhance the cluster by implementing auto-unseal using the transit secrets engine hosted on a dedicated Vault container, eliminating manual key entry. For simplicity, we use transit, but production environments should use a Key Management Service (KMS) (e.g., AWS KMS, Azure Key Vault) or Hardware Security Module (HSM) for enhanced security. This intermediate-to-advanced guide builds on Parts 1-3, delivering a production-like setup.

Prerequisites

Before starting, ensure you have:

Docker (version 20.10 or later). Install from Docker’s guide.
Docker Compose (version 2.0 or later). See Docker Compose installation.
Vault CLI. Download from releases.hashicorp.com/vault or use a package manager (e.g., brew install vault on macOS).
jq for parsing JSON (optional, for token extraction). Install via brew install jq (macOS) or equivalent.
A text editor (e.g., VS Code).
Familiarity with Parts 1-3 (development mode, persistent storage, Raft clustering with retry_join).
Commands are tested on Linux/macOS; Windows users may need to use set instead of export.

Reviewing the Cluster with Manual Unseal

In Part 3, you set up a three-node cluster (vault1, vault2, vault3) using Raft storage for HA, with retry_join for automated node discovery and separate volumes (vault1-data, vault2-data, vault3-data) to support Raft’s independent storage needs. As explained in Part 3, this transitioned from Part 2’s shared file storage to enable clustering. Each node requires manual unsealing with a key, which ensures security but is time-consuming. Let’s verify the setup before transitioning to auto-unseal.

Diagram

graph TD
    A[Client: Browser/CLI] --> B[Docker Compose]
    B --> C[Vault Node 1: Port 8200<br>Manual Unseal]
    B --> D[Vault Node 2: Port 8201<br>Manual Unseal]
    B --> E[Vault Node 3: Port 8202<br>Manual Unseal]
    C --> F[Vault1 Data]
    D --> G[Vault2 Data]
    E --> H[Vault3 Data]

Step 1: Verify the Existing Cluster

Assuming you have the Part 3 setup, verify the cluster:

Navigate to vault-cluster and check containers:
```
 docker ps
```
You should see vault1, vault2, and vault3.

Check cluster status on vault1:

 export VAULT_ADDR=http://localhost:8200
 export VAULT_TOKEN=<root-token-from-part3>
 vault operator raft list-peers

Sample Response:

 Node      Address          State       Voter
 ----      -------          -----       -----
 vault1    vault1:8201      leader      true
 vault2    vault2:8201      follower    true
 vault3    vault3:8201      follower    true

Test manual unseal by restarting:

 docker-compose down
 docker-compose up -d

Unseal each node (using keys from Part 3):

 export VAULT_ADDR=http://localhost:8200
 vault operator unseal <vault1-unseal-key>
 export VAULT_ADDR=http://localhost:8201
 vault operator unseal <vault2-unseal-key>
 export VAULT_ADDR=http://localhost:8202
 vault operator unseal <vault3-unseal-key>

Verify a secret:

 export VAULT_ADDR=http://localhost:8200
 vault kv get secret/my-app

This manual process underscores the need for automation.

Transit-Based Auto-Unseal with Dedicated Vault

Manual unsealing is tedious, time-consuming, and prone to errors, especially in a production environment where downtime matters. Auto-unseal offers a better way by automating the process, making Vault startup faster and more reliable. Now, we’ll introduce a completely separate vault-transit container to host the transit secrets engine, enabling auto-unseal for vault1, vault2, and vault3. This standalone vault-transit instance ensures isolation from the cluster. We’ll set up vault-transit first, then reconfigure the cluster to use auto-unseal, eliminating manual key entry.

Diagram

graph TD
    A[Client: Browser/CLI] --> B[Docker Compose]
    B --> C[Vault Node 1: Port 8200<br>Transit Auto-Unseal]
    B --> D[Vault Node 2: Port 8201<br>Transit Auto-Unseal]
    B --> E[Vault Node 3: Port 8202<br>Transit Auto-Unseal]
    B --> F[Vault-Transit: Port 8203<br>Transit Engine]
    C --> G[Vault1 Data]
    D --> H[Vault2 Data]
    E --> I[Vault3 Data]
    F --> J[Vault-Transit Data]
    C --> F[Transit Auto-Unseal]
    D --> F[Transit Auto-Unseal]
    E --> F[Transit Auto-Unseal]

Step 2: Update Docker Compose

Update docker-compose.yml to add vault-transit, keeping separate volumes:

services:
  vault1:
    image: hashicorp/vault:latest
    container_name: vault1
    ports:
      - "8200:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault1-data:/vault/file
      - ./vault-config/vault1:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault2:
    image: hashicorp/vault:latest
    container_name: vault2
    ports:
      - "8201:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault2-data:/vault/file
      - ./vault-config/vault2:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault3:
    image: hashicorp/vault:latest
    container_name: vault3
    ports:
      - "8202:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault3-data:/vault/file
      - ./vault-config/vault3:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault-transit:
    image: hashicorp/vault:latest
    container_name: vault-transit
    ports:
      - "8203:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-transit-data:/vault/file
      - ./vault-config/vault-transit:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

volumes:
  vault1-data:
  vault2-data:
  vault3-data:
  vault-transit-data:

networks:
  vault-net:
    driver: bridge

Key Notes:

Retains Part 3’s separate volumes (vault1-data, vault2-data, vault3-data) for Raft.
Adds vault-transit-data for vault-transit.
vault-net ensures communication.

Step 3: Configure Transit Vault

Create a vault-transit subdirectory in vault-config and add vault.hcl:

vault-config/vault-transit/vault.hcl:

 ui = true
 api_addr = "http://vault-transit:8200"

 storage "file" {
   path = "/vault/file"
 }

 listener "tcp" {
   address = "0.0.0.0:8200"
   tls_disable = 1
 }

Start containers:
```
 docker-compose up -d
```

Initialize vault-transit:

 export VAULT_ADDR=http://localhost:8203
 vault operator init -key-shares=1 -key-threshold=1

Sample Response:

 Unseal Key 1: <transit-unseal-key>
 Initial Root Token: s.<transit-root-token>

 Vault initialized with 1 key shares and a key threshold of 1.
 Please securely distribute the key shares printed above.

Unseal vault-transit:

 vault operator unseal <transit-unseal-key>

 export VAULT_TOKEN=<transit-root-token>
 vault login

Enable transit engine:

 vault secrets enable -path=transit transit

Sample Response:

 Success! Enabled the transit secrets engine at: transit/

Create an unseal key:

 vault write -f transit/keys/unseal_key

Sample Response:

 Success! Data written to: transit/keys/unseal_key

Create a policy:

 vault policy write transit-unseal - <<EOF
 path "transit/encrypt/unseal_key" {
   capabilities = ["update"]
 }
 path "transit/decrypt/unseal_key" {
   capabilities = ["update"]
 }
 EOF

Create a transit token:

 vault token create -policy=transit-unseal -format=json | jq -r .auth.client_token > transit-token.txt

Sample Response:

 {
   "auth": {
     "client_token": "<transit-token>",
     "policies": ["default", "transit-unseal"],
     "lease_duration": 2764800,
     ...
   }
 }

Step 4: Update Cluster for Auto-Unseal

Update vault1/vault.hcl, vault2/vault.hcl, and vault3/vault.hcl to add the seal stanza with the transit token from transit-token.txt:

vault-config/vault1/vault.hcl:

 ui = true
 api_addr = "http://vault1:8200"
 cluster_addr = "http://vault1:8201"

 storage "raft" {
   path = "/vault/file"
   node_id = "vault1"
   retry_join {
     leader_api_addr = "http://vault2:8200"
     leader_api_addr = "http://vault3:8200"
   }
 }

 listener "tcp" {
   address = "0.0.0.0:8200"
   tls_disable = 1
 }

 seal "transit" {
   address = "http://vault-transit:8200"
   token = "<transit-token>"
   mount_path = "transit/"
 }

vault-config/vault2/vault.hcl:

 ui = true
 api_addr = "http://vault2:8200"
 cluster_addr = "http://vault2:8201"

 storage "raft" {
   path = "/vault/file"
   node_id = "vault2"
   retry_join {
     leader_api_addr = "http://vault1:8200"
     leader_api_addr = "http://vault3:8200"
   }
 }

 listener "tcp" {
   address = "0.0.0.0:8200"
   tls_disable = 1
 }

 seal "transit" {
   address = "http://vault-transit:8200"
   token = "<transit-token>"
   mount_path = "transit/"
 }

vault-config/vault3/vault.hcl:

 ui = true
 api_addr = "http://vault3:8200"
 cluster_addr = "http://vault3:8201"

 storage "raft" {
   path = "/vault/file"
   node_id = "vault3"
   retry_join {
     leader_api_addr = "http://vault1:8200"
     leader_api_addr = "http://vault2:8200"
   }
 }

 listener "tcp" {
   address = "0.0.0.0:8200"
   tls_disable = 1
 }

 seal "transit" {
   address = "http://vault-transit:8200"
   token = "<transit-token>"
   mount_path = "transit/"
 }

Restart cluster nodes:

 docker-compose restart vault1 vault2 vault3

Initialize cluster nodes:
```
 export VAULT_ADDR=http://localhost:8200
 vault operator init -key-shares=1 -key-threshold=1
```
Sample Response:
```
 Success! Vault is initialized
 Initial Root Token: <cluster-root-token>

 Vault initialized with 1 key shares and a key threshold of 1.
 Please securely distribute the key shares printed above.
```
If you’re continuing from Part 3, you might wonder why reinitialization is needed. Switching from manual unsealing to transit-based auto-unseal requires a new keyring compatible with the transit secrets engine, as the existing Raft data is tied to manual unseal keys. Reinitializing applies the new seal configuration while preserving your data in the Raft storage volumes (vault1-data, vault2-data, vault3-data). Initializing one node (e.g., vault1) is sufficient, as Raft replicates the state to vault2 and vault3 via retry_join. All nodes auto-unseal automatically using vault-transit, so no unseal keys or additional initialization is needed.

Step 5: Test the Cluster

 export VAULT_ADDR=http://localhost:8200
 export VAULT_TOKEN=<cluster-root-token>
 vault login

Store a secret:

 vault kv put secret/my-app db_password=supersecret

Retrieve from vault2:

 export VAULT_ADDR=http://localhost:8201
 vault kv get secret/my-app

Check cluster status:

 vault operator raft list-peers

Sample Response:

 Node      Address          State       Voter
 ----      -------          -----       -----
 vault1    vault1:8201      leader      true
 vault2    vault2:8201      follower    true
 vault3    vault3:8201      follower    true

Step 6: Access Vault

Web UI: Open http://localhost:8200, 8201, or 8202 for the cluster; 8203 for vault-transit. Use the cluster or transit root token, respectively.
CLI: Use any cluster node’s address (e.g., VAULT_ADDR=http://localhost:8200).
Secrets persist via separate Raft storage volumes.

Step 7: Stop the Cluster

docker-compose down

On restart, unseal vault-transit manually; cluster nodes auto-unseal.

Best Practices

Use KMS or HSM in Production: Transit is for learning. Use a KMS (e.g., AWS KMS, Google Cloud KMS) or HSM for auto-unseal. See Vault Auto-Unseal Docs.
Secure Tokens: Store transit and root tokens in a password manager or HSM. Rotate transit tokens:
```
  vault token revoke <transit-token>
  vault token create -policy=transit-unseal
```
Multiple Keys in Production: Use multiple unseal keys (e.g., three out of five) for vault-transit.
Enable TLS: Set tls_disable = 0 and use certificates.
Monitor Raft and Retry Join: Ensure Raft storage is backed up and retry_join targets are stable.
Backup Storage: Back up vault1-data, vault2-data, vault3-data, and vault-transit-data volumes.
Isolate Transit Vault: The separate vault-transit instance enhances security.

Troubleshooting

Retry Join Fails: Verify target nodes (curl http://vault1:8200/v1/sys/health). Check vault-net (docker network inspect vault-net).
Auto-Unseal Fails: Ensure vault-transit is unsealed and the transit token is valid. Check logs (docker logs vault1).
Raft Issues: Verify separate volumes (docker inspect vault1). Check logs (docker logs vault1).
Token Errors: Confirm the transit token matches transit-token.txt and has the transit-unseal policy.

What’s Next?

You’ve transitioned your Vault cluster to transit-based auto-unseal! To extend your learning:

Adapt MySQL dynamic secrets from Part 2.
Test leader failover by stopping vault1 and checking vault operator raft list-peers.
Explore KMS-based auto-unseal using Vault Auto-Unseal Docs.

Share your progress in the comments or join the HashiCorp Community Forum!

Resources

HashiCorp Vault Documentation: vaultproject.io
Vault Auto-Unseal Docs: vaultproject.io/docs/configuration/seal
Vault HA Documentation: vaultproject.io/docs/concepts/ha
Docker Compose Documentation: docs.docker.com/compose

Note: For questions or setup help, comment below or check the Vault documentation.

HashiCorp Vault with Docker Compose: Advanced Clustering and Auto-Unseal (Part 4)

HashiCorp Vault with Docker Compose: Advanced Auto-Unseal with Transit (Part 4)

Prerequisites

Reviewing the Cluster with Manual Unseal

Diagram

Step 1: Verify the Existing Cluster

Transit-Based Auto-Unseal with Dedicated Vault

Diagram

Step 2: Update Docker Compose

Step 3: Configure Transit Vault

Step 4: Update Cluster for Auto-Unseal

Step 5: Test the Cluster

Step 6: Access Vault

Step 7: Stop the Cluster

Best Practices

Troubleshooting

What’s Next?

Resources

Subscribe to my newsletter

Miroslav Milak

Miroslav Milak