Services require access keys, passwords, and certificates to interact with other services ( internal or external ). It's possible to store passwords or access keys as environmental variables on computing servers executing/running these services and assign appropriate permissions to restrict access control. This approach is suitable for small-medium corporate networks.

However, in large corporate networks, I advise not to use this approach because of: the difficulty in managing secret keys, and inefficient secret auditing. It's advisable to centralize storage for managing secrets due to the following benefits: transparency in secrets management and effective secret auditing.

In this article, I share my experience on how to threat model Hashicorp Vault to expose different ways hackers would attack and discuss preventive solutions to mitigate attacks against the Vault server.

Check this documentation for detailed information on Vault, how it works, etc.

Securing Infrastructure

The following discusses how attackers could shut down the server hosting the Vault service.

DDOS Attack

DDOS ( distributed denial of service) is a common technique for hackers to prevent services from responding to legitimate requests by sending excessive requests to the server. These requests initiated by bots are programmed to send requests from the internet to your corporate network.

DDOS Attack Solutions

Rate Limit Request: Build/Develop a solution/feature to rate limit auth requests to the Vault service. For instance, you can define a constraint policy to restrict clients for about 30 seconds or a minute after three failed authentication attempts.
Blacklist IP addresses: I have never recommended clients use this approach to prevent DDOS attacks because hackers use DHCP servers to manage IP addresses. However, you can temporarily blacklist suspicious IP addresses submitting Auth requests after previous successful Auth requests or scanning your servers continuously.

Network Downtime:

Apart from using DDOS to make servers inaccessible, network downtime is another event to consider as we threat model a key vault server. Power outages can prevent clients/users from connecting to the Vault server. This situation could also affect the business's reputation.

Network Downtime Solutions

Geo-Replicate Servers: You can deploy a replica of the Vault server in another region to failover/takeover once the main/primary server hosting Vault is not available to serve requests.

NB: Automate server failover ( with scripting languages like Python, or Bash) instead of doing it manually to avoid increasing downtime.

External Power System: Another solution to consider is an external power system to supply power to the Vault server. You can rely on UPS devices to "buy time" for resolving power outages - assuming it's an internal technical issue.

Corrupt Server Disks:

Malicious users sometimes use malware to corrupt server disks to prevent enterprises from accessing business data. Loss of business data affects a business' reputation and incurs charges for violating data privacy regulations.
Since the Hashicorp Vault service persists secrets to disk, we need a solution to counteract disk corruption.

Disk Corruption Solution:

Some Linux operating systems like Tails OS provides a disk encryption solution ( Luks encryption) to encrypt the disk during the operating system booting stage. I recommend Tails OS or other Linux operating systems with disk encryption solutions to prevent disk corruption issues.

Securing Hashicorp Vault Service

Hashicorp Vault consists of the following components: Audit device, Secret Engines, Auth Method, Policy Store, and Token Store. These components work together ( but serve different purposes) to secure your secrets.

In our quest to secure Vault from unauthorized users/services, let's discuss different ways to configure or set up these components - secret engines, auth method, and audit device - to mitigate attacks from malicious users.

Access Control Management

Hashicorp Vault provides various ways - CLI, UI, HTTP - for clients/users to connect to the server to generate tokens or create policies. Thus, the first thing to look at ( when threat modeling) is to prevent or restrict access to the Vault server.

Access Control for Services/Users/Admin

For Web Services:

Create allow rule policy for all services, users, and admins to have access to the Vault server. If you choose to create allow rule policy, you need to ensure that hosts/servers where your services run are not managed by DHCP servers. On the other hand, if your services are managed by a DHCP server, then you can use make use of OPA (open policy agent) to create policies based on hostname instead of IP addresses - to avoid editing IP addresses.

For Users/Admin:

Instead of allowing users/admins to connect to the Vault server directly, I advise directing users to a Bastion host via SSH keys. Then users/admin can connect from the bastion host to the Vault server.

NB: Why a bastion host?

A Bastion host usually prevents direct access to services in a corporate network. Some companies have also leveraged this to create custom conditional access solutions to verify users apart from SSH keys - because a user's SSH keys are not safe from a malicious keylogger or rootkit.

Securing External Auth Method (Authentication)

This component ensures that users/services are verified before access is granted. Hashicorp Vault supports both in-built and external auth methods.

Most enterprises use diverse authentication systems like Microsoft Azure ( for both cloud and on-premise workloads), Kerberos, Okta, etc. So you need to apply appropriate measures to secure your authentication systems.

NB: In my next article, I will share practical tips on securing a Kerberos system in a Unix environment. Kerberos KDC (Key Distribution Center) is one important component you need to protect at all costs. Make sure the server hosting Kerberos KDC is not critically vulnerable.

Securing Secret Engines

Secret engines generate, store, and encrypt data or otherwise known as secrets. However, some secret engines differ in function. Some secret engines only generate and store whilst others do not only generate and store but encrypt data/secrets.

Hashicorp Vault provides support for the following secret engines.

So let's assume we have a couple of web services using a Postgresql database as their backend system. Each web service stores data in a separate Postgres table and requires credentials to access the Postgres database.

These services do not need to hardcode Postgres credentials but access Vault to retrieve data from Postgres tables. So we need to ensure that only legitimate services have access to the server hosting the vault service to generate secrets for services/users.

Restricting Access to pg_hba.conf File

Although Vault manages Postgres credentials, we need to ensure that the pg_hba.conf file is not accessible by unauthorized users/services. If the pg_hba.conf file is publicly accessible, then malicious users could change Postgres authentication mode to "Trust authentication" mode.

In this case, the Vault server becomes useless even if it generates credentials for Postgres roles. Also, the Postgres server should not be publicly accessible.

Securing Audit Devices

Vault auditing devices record requests and responses to the Vault server. So we need to secure audit devices to monitor interactions between the Vault server and other services/users.

As of the time of writing, Hashicorp Vault writes logs to local syslog (not remote log server), file and TCP, UDP, and UNIX sockets.

You can configure Vault to write logs to Syslog. However, it's advisable to stream logs from Syslog to a remote central log system like Elasticsearch where we can analyze interactions between clients and Vault.

Finally, we need to monitor the Vault instance or host - with Apache Falco or another detection tool - to detect and alert the SOC team anytime a malicious software/user tries to rotate Syslog to hide malicious activities or tamper with Vault configuration settings.

Threat Modeling Key Management Service (Hashicorp Vault Opensource)