Encrypting and crypto-shredding of PII data in Marten documents using HashiCorp Vault

Protecting Personally Identifiable Information (PII) has become essential as organizations handle increasing volumes of sensitive data. Regulations like the GDPR (General Data Protection Regulation) mandate strict controls over PII to safeguard individuals' privacy and prevent data breaches. One effective approach to securing PII is data masking, a method that conceals sensitive information by replacing it with altered values, rendering it useless if accessed by unauthorized parties. Masking techniques such as character substitution, encryption, and tokenization are also used to protect data.

I have written about masking by characters substitution and encryption using a standard global key across Marten documents.

This blog post is a continuation of the encryption mechanism and will focus on using Hashicorp Vault backends, with granular per-document encryption key support. Also do crypto-shredding i.e. deliberately deleting or overwriting the encryption key which renders the encrypted data irrecoverable. It is important to read encryption using a standard global key across documents article which is a prerequisite for this blog post.

As a first step, let us look at the VaultEncryptionService implementation which uses the HashiCorp Vault for encryption.

using VaultSharp;  
using VaultSharp.V1.AuthMethods.Token;  
using VaultSharp.V1.SecretsEngines.Transit;  

namespace marten_docs_pii;  

public class VaultEncryptionService: IEncryptionService  
{  
    private readonly VaultClient _client;  
    private const string DefaultKeyName = "pii-key";  

    public VaultEncryptionService(string vaultAddress, string token)  
    {        var authMethod = new TokenAuthMethodInfo(token);  
        var vaultClientSettings = new VaultClientSettings(vaultAddress, authMethod);  
        _client = new VaultClient(vaultClientSettings);  
    }    public async Task<string> EncryptAsync(string plainText, string? key = null)  
    {        key ??= DefaultKeyName;  
        var result = await _client.V1.Secrets.Transit.EncryptAsync(  
            key,            new EncryptRequestOptions  
            {  
                Base64EncodedPlainText = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(plainText))  
            });            return result.Data.CipherText;  
    }  
    public async Task<(bool success, string plainText)> TryDecryptAsync(string cipherText, string? key = null)  
    {        var (success, plainText) = await TryDecryptInternalAsync(cipherText, key);  
        return (success, plainText);  
    }  
    private async Task<(bool success, string plainText)> TryDecryptInternalAsync(string cipherText, string? key = null)  
    {        try  
        {  
            key ??= DefaultKeyName;  
            var result = await _client.V1.Secrets.Transit.DecryptAsync(  
                key,                new DecryptRequestOptions { CipherText = cipherText });  

            var decryptedBytes = Convert.FromBase64String(result.Data.Base64EncodedPlainText);  
            return (true, System.Text.Encoding.UTF8.GetString(decryptedBytes));  
        }        catch  
        {  
            return (false, string.Empty);  
        }    }  
    public async Task DropEncryptionKeyAsync(string key)  
    {        if (string.IsNullOrWhiteSpace(key))  
        {            return;  
        }        await _client.V1.Secrets.Transit.UpdateEncryptionKeyConfigAsync(key, new UpdateKeyRequestOptions  
        {  
            DeletionAllowed = true  
        });  

        await _client.V1.Secrets.Transit.DeleteEncryptionKeyAsync(key);  
    }}

This class provides encryption and decryption capabilities using HashiCorp Vault's Transit Secrets Engine. The service implements an IEncryptionService interface and uses the VaultSharp client library to interact with Vault.

The service is initialized with a Vault address and authentication token, and uses a default encryption key named pii-key. It provides three main operations:

1. EncryptAsync: Encrypts plaintext by first converting it to Base64, then using Vault's Transit engine to perform the encryption, returning the resulting ciphertext.

2. TryDecryptAsync: Attempts to decrypt ciphertext using the specified key (or default key). It handles decryption failures gracefully by returning a tuple containing a success flag and the decrypted text (or empty string if decryption fails).

3. DropEncryptionKeyAsync: Provides functionality to delete an encryption key from Vault, first enabling deletion permission and then performing the actual deletion.

When key is passed, it will used for both encryption and decryption which becomes document specific key. Otherwise, it uses the default encryption key which can be used across documents. Note that we are keeping the encryption keys secure in Vault rather than in the application itself to enhance its robustness and security.

Identification of document specific encryption key

Each document will need to implement the marker interface IHasEncryptionKey as below

public interface IHasEncryptionKey {
    string EncryptionKey { get; }
}

By way of implementing the interface, EncryptionKey will need to be set for each document instance. EncryptionKey will get used as the key for encryption in Vault for the properties pertaining to the document.

Let us look at how to add IHasEncryptionKey to Person record as below:

public record Address(string Street, string City);

public record Person(Guid Id, string Name, string Phone, Address Address) : IHasEncryptionKey
{
    public string EncryptionKey => Id.ToString();
}

In the above code, EncryptionKey is set with Id as string.

Let us understand how this EncryptionKey is fetched in TransformDocumentAsync in EncryptionRules for passing it for encryption related functions.

string? key = null;

// check if document implement IHasEncryptionKey
if (documentType.GetInterfaces().Any(x => x == typeof(IHasEncryptionKey)))
{
    key = ((IHasEncryptionKey)document).EncryptionKey;
}

Call await _encryptionService.EncryptAsync(currentValue, key); or await _encryptionService.TryDecryptAsync(currentValue, key) passing the key.

The following is the docker-compose file to run Vault as a docker container in your development environment:

services:
  vault:
    image: hashicorp/vault:latest
    container_name: vault
    ports:
      - "8300:8300"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"
      VAULT_LOCAL_CONFIG: |
        {
          "listener": {
            "tcp": {
              "address": "0.0.0.0:8200",
              "tls_disable": 1
            }
          },
          "backend": {
            "file": {
              "path": "/vault/data"
            }
          },
          "default_lease_ttl": "168h",
          "max_lease_ttl": "720h",
          "ui": true
        }
    volumes:
      - ./vault-data:/vault/data
    cap_add:
      - IPC_LOCK
    restart: unless-stopped

Note that we have configured Vault to run as a HTTP service instead of HTTPS in dev.

Now let us put all of these pieces together with an example usage. As a firsts step, create an instance of `VaultEncryptionService` as below:

```csharp
var encryptionService = 
    new VaultEncryptionService("http://localhost:8200", "root");

We are passing the Vault URL and token as a constructor.

As a next step, create the document store including setting up the encryption rules:

await using var store = DocumentStore.For(opts =>
{
    opts.Connection(
"Host=localhost;Database=marten_testing;Username=postgres;Password=postgres");
    opts.UseEncryptionRulesForProtectedInformation(encryptionService);
    opts.Schema.For<Person>()
        .AddEncryptionRuleForProtectedInformation(x => x.Name)
        .AddEncryptionRuleForProtectedInformation(x => x.Phone)
        .AddEncryptionRuleForProtectedInformation(x => x.Address.Street);
});

This code snippet demonstrates the configuration of Marten's DocumentStore with encryption capabilities for protecting sensitive data. The configuration uses a fluent API to set up both database connection and encryption rules. First, it establishes a connection to a PostgreSQL database using standard connection parameters. Then, through UseEncryptionRulesForProtectedInformation, it integrates the encryption service (VaultEncryptionService in this case) into Marten's pipeline.

The most interesting part is the schema configuration for the Person class, where it explicitly defines which properties should be encrypted using AddEncryptionRuleForProtectedInformation. In this case, it marks Name, Phone, and the nested Address.Street properties for encryption, demonstrating the system's ability to handle both top-level and nested property encryption.

Let us add a document and inspect the data in database as below:

await using var session = store.LightweightSession();

// Create and store a person
var personId = Guid.NewGuid();
var person1 = new Person(
    personId, 
    "John Doe", 
    "111-111", 
    new Address("123 Main St", "Anytown"));

session.Store(person1);
await session.SaveChangesAsync();

When you inspect the "data at rest" as stored in database, you will see the below with the right set of properties encrypted via Vault i.e Name, Phone and Address.City:

{
    "Id": "ecd0be32-08d9-46f6-be03-a12355b8ab8a",
    "Name": "vault:v1:h+y3qfgNEJ5IlldaUpM2yH2ZPoosQwP8Ecqrim2VqkSq5cgi",
    "Phone": "vault:v1:7gHToyLraxeC4bX0eFWLedjDsrBWDKZv/2lqcuvolYS8Be4=",
    "Address": {
        "City": "Anytown",
        "Street": "vault:v1:fsfACkhaXkqrPCPlaoYY+KN6B4uHvnbjGQfbsRzz5IIMTouRAynG"
    },
    "EncryptionKey": "ecd0be32-08d9-46f6-be03-a12355b8ab8a"
}

If you retrieve the document using Marten session, you will see that the data is decrypted properly as below:

var person = await session.LoadAsync<Person>(personId);  
Console.WriteLine($"Name: {person?.Name}"); // Will show decrypted value  
Console.WriteLine($"Phone: {person?.Phone}"); // Will show decrypted value  
Console.WriteLine($"Street: {person?.Address.Street}, City: {person?.Address.City}");

You will see the output as below:

Name: John Doe
Phone: 111-111
Street: 123 Main St, City: Anytown
Name: John Doe, Phone: 111-111, Street: 123 Main St

Crypto-shredding

Crypto-shredding is the practice of rendering encrypted data unusable by deliberately deleting or overwriting the encryption keys, the data should become irrecoverable, effectively permanently deleted or "shredded".

To achieve this, we will do the following by calling DropEncryptionKeyAsync:

await encryptionService.DropEncryptionKeyAsync(personId);

var person = await session.LoadAsync<Person>(personId);  
Console.WriteLine($"Name: {person?.Name}"); // Will show decrypted value  
Console.WriteLine($"Phone: {person?.Phone}"); // Will show decrypted value  
Console.WriteLine($"Street: {person?.Address.Street}, City: {person?.Address.City}");

Now the output is:

Name: vault:v1:6uIYFOZC8jsYRrpN3WQyGfVgRiPbW5WTPmBuTDs9lxTVnqFX
Phone: vault:v1:XtVUMw9EKwtBV655AazcbHAFs/Th9FCM/lVcr/F+ChAjmQQ=
Street: vault:v1:BnErhi9JNvRPUvKQYh0y2fNyGLj3zAUX7vIXJSrDaKF1rG5zH2I0, City: Anytown
Name: vault:v1:6uIYFOZC8jsYRrpN3WQyGfVgRiPbW5WTPmBuTDs9lxTVnqFX, Phone: vault:v1:XtVUMw9EKwtBV655AazcbHAFs/Th9FCM/lVcr/F+ChAjmQQ=, Street: vault:v1:BnErhi9JNvRPUvKQYh0y2fNyGLj3zAUX7vIXJSrDaKF1rG5zH2I0

Notice that the encrypted data is returned "as is" since the decryption logic is not able to decrypt it. For the data itself, you could choose to do characters substitution like ****, ###-### etc use or combine functionality as explained in blog post masking by characters substitution

In summary, this implementation demonstrates document encryption using Marten with HashiCorp Vault with encryption per document. The code showcases a complete workflow for storing and retrieving encrypted documents. The Marten DocumentStore is configured with encryption rules that specifically target sensitive fields in the Person record: Name, Phone, and the nested Address.Street property. The program creates a single Person instance with sample data and demonstrates the transparent encryption/decryption process by storing and then retrieving the document. After saving, it verifies the encryption by loading the document back and displaying the automatically decrypted values. The Person class is implemented as a simple record implementing IHasEncryptionKey interface, meaning it uses per-document keys.

The source is available here for your ready reference. Happy coding :-)