Understanding the Seal and Unseal Process in HashiCorp Vault
Table of contents
- What Happens When Vault is Sealed?
- Key Concepts of Vault Sealing
- The Encryption Hierarchy
- What Does Unsealing Vault Mean?
- Why is Sealing Vault Important?
- Ways to Unseal Vault
- The Unseal Process
- Recovery Keys
- Sealing and Resealing Vault
- Steps for Initialization and Rekeying
- Seal Migration
- High Availability (Seal HA)
- Best Practices for Seal Management
- Conclusion
- References
When learning about HashiCorp Vault, one important topic to understand is the seal and unseal process. This process is a big part of Vault’s security system. When Vault is initialized, it begins in a sealed state. This critical stage ensures that Vault is aware of its physical storage location but does not have the decryption key necessary to access the data. This is because the key needed to unlock the data is not available until the unseal process happens. Unsealing is the process that unlocks the necessary keys for Vault to function.
What Happens When Vault is Sealed?
When Vault is sealed, it can do very limited operations. The only actions allowed are checking the status to see if Vault is sealed or unsealed and starting the unseal process. In a sealed state:
Apps cannot get any data.
Vault cannot create or unlock data.
Encrypted data inside Vault stays locked.
This means that a sealed Vault is not useful for most regular activities until it is unsealed.
Key Concepts of Vault Sealing
Vault operates with a multi-layered encryption approach:
Encrypted Data: All data stored within Vault is encrypted.
Encryption Key: This is stored in the keyring and is required for data decryption.
Root Key: Used to encrypt the keyring.
Unseal Key: Encrypts the root key.
The Encryption Hierarchy
Data Encryption Key: Vault encrypts data using a unique key stored in its keyring.
Keyring Encryption: The keyring is protected by a root key.
Root Key Encryption: This key is encrypted by an unseal key.
Simply put, to access the stored data, Vault must decrypt the keyring using the root key, which in turn requires the unseal key.
What Does Unsealing Vault Mean?
Unsealing Vault is very important for making it work. During unsealing, a node rebuilds the master key, which is then used to get the encryption key. This key is needed to read and write data. Once unsealed, Vault keeps this key in memory so it can handle data easily. Each node in a group needs to be unsealed separately because each one keeps the encryption key on its own.
Why is Sealing Vault Important?
Sealing Vault is an important safety step. When sealed, Vault deletes the encryption key from memory, so unsealing is needed to use Vault again. This helps keep data safe during problems or security risks.
Common Reasons to Manually Seal Vault:
Key shards are exposed: If parts of the master key are shared by mistake, like on GitHub.
Employee changes: When someone who had key parts leaves the company.
Network issues: If there is a possible or confirmed break-in, Vault should be sealed to protect data.
Malware or spyware: If these are found on Vault servers, sealing helps keep data secure.
Ways to Unseal Vault
There are different ways to seal and unseal Vault:
Key Sharding with Shamir’s Secret Sharing Algorithm (Default Way):
- The master key is split into parts, and enough parts need to come together to unseal Vault.
Cloud Auto-Unseal:
- Uses cloud services like AWS Key Management Service (KMS) or Azure Key Vault to unseal Vault automatically. This makes things faster and easier, with less need for people to do it.
Transit Auto-Unseal:
- Uses another Vault cluster to unseal nodes, making the process easier for multiple Vault servers.
The Unseal Process
Unsealing is essential to gain access to the root key, which decrypts the encryption key and thus unlocks Vault. Before unsealing, no operational tasks, such as authentication or secret retrieval, are possible.
Steps for Unsealing
Initiate Unseal Process: Run
vault operator unseal
or use the Vault API. Operators manually provide shares until the threshold is met:$ vault operator unseal
Enter Unseal Keys: Input the unseal key shares one at a time. These can be provided from different client machines for enhanced security.
Reconstruct the Root Key: Once the threshold number of shares is entered, the root key is reconstructed, decrypting the encryption key and unsealing Vault.
Shamir's Secret Sharing
Vault’s default configuration uses Shamir's Secret Sharing to split the unseal key into multiple shares, requiring a defined threshold of these shares to reconstruct the key. This improves security by distributing key shares across different trusted operators.
Important Note: Each Vault node in a cluster must be unsealed individually. Partial unsealing is not distributed across nodes.
Benefits of Shamir’s Secret Sharing
Enhanced Security: No single person holds the complete key.
Distributed Control: Multiple operators are needed for unsealing.
Limitations of Shamir's Seal
Manual Effort: The process is not automated, making it cumbersome for frequent restarts.
Cluster Constraints: Each node must be unsealed independently, as partial unsealing is not distributed.
Auto Unseal Mechanism
Auto Unseal is designed to simplify the unsealing process, particularly for automated deployments. It leverages trusted external devices or services, such as HSMs (Hardware Security Modules) or cloud-based KMS (Key Management Services), to securely store and decrypt the unseal key at startup.
How Auto Unseal Works:
Vault Startup: Vault connects to the configured device or service.
Request Decryption: The external mechanism decrypts the stored root key.
Automatic Unseal: The root key decrypts the encryption key, completing the unseal process without manual key entry.
Advantages of Auto Unseal:
Operational Efficiency: Eliminates manual unsealing.
Security: Offloads key protection to trusted external systems.
Auto Unseal Dependencies
However, using auto unseal creates a strict dependency on the external service. If this service becomes unavailable, Vault cannot be accessed even with backups. It’s crucial to maintain rigorous controls over this dependency.
Recovery Keys
When using Auto Unseal, Vault returns recovery keys instead of unseal keys during initialization. The main difference between unseal keys and recovery keys lies in their functionality:
Unseal Keys: Used specifically to decrypt the root key and unseal Vault. Without these, the unsealing process cannot proceed.
Recovery Keys: Serve as an authorization mechanism for critical tasks, such as generating a root token, but they cannot decrypt the root key. This makes them insufficient for unsealing Vault if the auto-unseal feature fails.
Recovery Key Operations:
Generate-Root: Requires a quorum of recovery keys.
Rekeying: Recovery keys can be rekeyed to alter the number of shares or change the key holders using different PGP keys.
Sealing and Resealing Vault
Vault can be resealed manually via the API, which discards the root key from memory and locks Vault. This action ensures that, in case of a detected intrusion, data access is immediately restricted.
Common Resealing Triggers:
Server Restart
Unrecoverable Storage Errors
Manual Reseal Command
Steps for Initialization and Rekeying
Initializing Vault
Initialization involves setting up shares and thresholds:
$ vault operator init -recovery-shares=5 -recovery-threshold=3
Rekeying Vault
Vault’s unseal key can be updated via:
$ vault operator rekey
The new key is wrapped by the HSM or KMS and stored securely.
Seal Migration
Migrating seals requires careful planning, as it often involves downtime. Seal migration is important for maintaining or upgrading the security posture of a Vault environment. The need for migration can arise from changes in organizational security policies, the introduction of more robust cryptographic methods, or the desire to leverage new features provided by different seal mechanisms. Ensuring that Vault uses the most appropriate and secure seal mechanism helps protect critical data and maintain operational resilience. Different procedures apply depending on the type of migration:
Migrating from Shamir to Auto Unseal:
Take the Cluster Offline.
Update Configuration: Add the auto-unseal configuration block.
Run the Unseal Command with the
-migrate
flag.Reconstruct Recovery Keys: Once complete, the unseal keys are migrated to recovery
Migrating from Auto Unseal to Shamir
Update Configuration: Disable the auto-unseal block (disabled = "true").
Use Recovery Keys: Enter recovery keys for the migration.
Run the Unseal Command: Execute with the
-migrate
flag.
Integrated Storage Considerations
For setups using Integrated Storage (Raft protocol):
Ensure quorum is reached after bringing nodes back online.
The leader node will perform the migration, with updates replicated across peers.
High Availability (Seal HA)
Seal High Availability (HA) allows configuration of multiple auto-seal mechanisms, enabling Vault to withstand temporary seal mechanism failures.
Seal HA Configuration
Choose Two to Three Seals: Ensure they are from independent sources (e.g., different cloud regions/providers).
Monitor Rewraps: When a seal configuration changes, Vault rewraps all seal-wrapped values.
Failover Capability: Vault can operate in a degraded mode if one seal is unavailable.
Limitations
Mixing Seal Types: Only auto seals can be used in HA.
Seal Wrapping Constraints: Large entries may face size limitations when wrapped by multiple seals.
Best Practices for Seal Management
Control Access: Implement strict controls for managing seal mechanisms, such as AWS Service Control Policies.
Monitor Configurations: Use the
/sys/sealwrap/rewrap
endpoint to track rewrap progress.Backup Regularly: Always take a backup before initiating seal migrations.
Plan for Downtime: Ensure minimal disruption by planning seal migration during low-traffic periods.
Conclusion
In conclusion, understanding the seal and unseal process in HashiCorp Vault is crucial for maintaining the security and functionality of our data management system. The seal process ensures that sensitive data remains protected by removing the encryption key from memory, while the unseal process is necessary to restore access and functionality. By leveraging different unseal methods, such as Shamir’s Secret Sharing and Auto Unseal, organizations can balance security with operational efficiency. Additionally, the ability to migrate between seal mechanisms and configure high availability options provides flexibility and resilience in managing Vault environments. Implementing best practices for seal management, such as controlling access and monitoring configurations, further enhances the security and reliability of Vault operations.
References
Subscribe to my newsletter
Read articles from Tanvir Sayyad directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
Tanvir Sayyad
Tanvir Sayyad
Hi there! I am a competent DevOps Engineer with 2 years of experience in the IT domain with tools and technologies diversity. My professional background encompasses Jenkins, Ansible, HashiCorp Vault, Bitbucket, GitHub, Nexus, Maven, Ant, and Docker. Some of the technologies that I'm familiar with are Terraform, Kubernetes, and GitLab CI/CD. My thirst for learning is never quenched, and in turn, I share my views on it through my blog. Let's connect and learn DevOps together! Feel free to reach out if you’re interested in collaboration or discussions about DevOps innovations at tanvir.sayyad1011@gmail.com.!