GitLab Pipeline JWT Authentication with HashiCorp Vault: A Comprehensive Guide

This guide explores how to integrate GitLab CI/CD pipelines with HashiCorp Vault using JWT (JSON Web Token) authentication to securely fetch secrets. We’ll cover three approaches—manual CI_JOB_JWT (older GitLab versions), manual id_tokens without secrets, and the modern secrets method—along with Vault setup, best practices, and testing tips. Whether you’re on GitLab 13.4 or the latest version, this guide has you covered.

Why Use JWT for Vault Authentication?

Using JWT for authentication allows GitLab runners to securely authenticate to Vault without embedding static credentials. The benefits include:

• Short-lived tokens: Reduces risk exposure.

• Automatic token rotation: No need to manually manage credentials.

• Granular access control: Vault policies can define fine-grained permissions based on GitLab project, branch, and environment.

---

Here’s a step-by-step guide to set up the KV Secrets Engine with GitLab:

What Are Vault Secrets Engines?

Vault Secrets Engines are components of HashiCorp Vault that manage secrets and sensitive data. Each engine is mounted at a specific path in Vault and provides an API to interact with secrets. The KV Secrets Engine is widely used to store static key-value pairs, such as API tokens or passwords, making it ideal for GitLab integration. In this setup, GitLab Pipelines can fetch these secrets securely during runtime.

GitLab’s Integration with the KV Secrets Engine

GitLab Pipelines integrates with HashiCorp Vault to retrieve secrets from the KV Secrets Engine using JSON Web Tokens (JWT) for authentication. Here’s the process:

• Authentication: GitLab generates a JWT for each pipeline job, which Vault verifies using GitLab’s JWKS endpoint (https://gitlab.com/-/jwks). If valid, Vault provides a temporary token.

• Secrets Access: The pipeline uses this token to fetch secrets from the KV Secrets Engine. This ensures secrets remain secure and are never exposed in your repository or CI variables.

---

Scenario

• GitLab: https://gitlab.example.com (self-hosted)

• Vault: https://vault.example.com

• Goal: Fetch an api_key secret for project ID 42, main branch

---

Step 1: Setting Up Vault

Vault must be configured to trust GitLab’s JWTs and grant access based on pipeline metadata.

1. Enable JWT Authentication

vault auth enable jwt

2. Configure the JWT Backend

Link Vault to your GitLab instance:

vault write auth/jwt/config \
    jwks_url="https://gitlab.example.com/-/jwks" \
    bound_issuer="https://gitlab.example.com" \
    default_role="project-42-role"

• jwks_url – GitLab’s public key endpoint for JWT verification.

• bound_issuer – Ensures JWTs come from your GitLab instance.

• default_role – Optional fallback role (override in pipeline if needed).

Note: For GitLab.com, use jwks_url="https://gitlab.com/-/jwks" and bound_issuer="https://gitlab.com".

3. Create a Role

Define a role to map GitLab pipeline metadata to Vault policies:

vault write auth/jwt/role/project-42-role \
    role_type="jwt" \
    policies="project-42-policy" \
    token_ttl="5m" \
    token_max_ttl="10m" \
    bound_claims.project_id="42" \
    bound_claims.ref="main" \
    bound_claims.ref_type="branch" \
    bound_claims.ref_protected="true"

bound_claims – Restricts access to project 42, main branch, and protected refs.

4. Create a Policy

Grant read access to specific secrets:

vault policy write project-42-policy - <<EOF
path "secret/data/project-42/*" {
    capabilities = ["read"]
}
EOF

5. Enable Secret Engine and Store a Secret

Enable the KV Secrets Engine in Vault:

vault secrets enable -path=secret kv-v2

version 2 is recommended for features like versioning

Add the secret to Vault:

vault kv put secret/project-42/api api_key="s3cr3t-k3y"

Note: Uses KV v2 (secret/data/) for versioning support.

6. Enable Audit Logging

Track secret access:

vault audit enable file file_path=/var/log/vault_audit.log

---

Step 2: GitLab Pipeline Configuration

GitLab offers multiple ways to authenticate with Vault using JWTs, depending on your version and needs.

Option 1: Manual Way with CI_JOB_JWT (GitLab 13.4-13.12)

For older GitLab versions, use CI_JOB_JWT manually:

deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  before_script:
    - apk add --no-cache vault
    - if [ -z "$CI_JOB_JWT" ]; then echo "JWT missing!" && exit 1; fi
  script:
    - export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=project-42-role jwt=$CI_JOB_JWT)
    - export API_KEY=$(vault kv get -field=api_key secret/project-42/api)
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production

How It Works: CI_JOB_JWT (introduced in 13.4) is sent to Vault to obtain a token, then used to fetch the secret.

• CI_JOB_JWT is a predefined variable in GitLab 13.4+ containing a signed JWT with pipeline metadata (e.g., project_id, ref).

• The script sends it to Vault’s auth/jwt/login endpoint, gets a Vault token, and uses it to fetch the secret.

• No secrets or id_tokens - all logic is manual.

Option 2: Manual Way with id_tokens (GitLab 14.0+)

Use id_tokens for a custom JWT ($VAULT_ID_TOKEN), handling Vault calls manually:

deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.example.com"
  before_script:
    - apk add --no-cache vault
    - if [ -z "$VAULT_ID_TOKEN" ]; then echo "ID token missing!" && exit 1; fi
  script:
    - export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=project-42-role jwt=$VAULT_ID_TOKEN)
    - export API_KEY=$(vault kv get -field=api_key secret/project-42/api)
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production

How It Works:

• ID_TOKENS defines VAULT_ID_TOKEN as a JWT with a custom aud claim (https://vault.example.com).

• The script manually sends this JWT to Vault’s auth/jwt/login endpoint to get a token, then fetches the secret.

• No secrets keyword—full control is in the script.

Option 3: Modern Way with secrets (GitLab 14.0+)

The streamlined approach using secrets:

deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.example.com"
  secrets:
    API_KEY:
      vault: secret/project-42/api/api_key
  script:
    - if [ -z "$API_KEY" ]; then echo "Secret not injected!" && exit 1; fi
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production

How It Works:

• id_tokens generates a JWT tailored for Vault (with a custom aud claim).

• secrets handles authentication and secret injection automatically.

Sidenote: Why id_tokens and secrets Are Preferred Over CI_JOB_JWT

• Security Improvements: id_tokens allow for specifying an audience (aud), reducing the risk of token misuse.

• Flexibility: id_tokens provide a cleaner integration with external authentication providers, allowing better access control.

• Modern Best Practices: The secrets method automates secret injection, reducing the need for manual script handling.

• Deprecation of CI_JOB_JWT: GitLab is phasing out CI_JOB_JWT in favor of id_tokens. Update your pipeline configurations accordingly!

---

Best Practices

Secure Variables

• Store VAULT_ADDR and VAULT_CACERT as protected, masked GitLab CI/CD variables in Settings > CI/CD > Variables.

• Mark them as protected to limit access to protected branches (e.g., main).

• Use masked variables for sensitive data to prevent accidental logging.

• Always use verified images

Restrict Pipeline Execution

• Use rules with $CI_COMMIT_REF_PROTECTED to limit secrets are only accessed in protected branches or tags.

• Avoid running on untrusted branches (e.g., feature branches) to prevent JWT misuse.

Environment Tracking

• Define an environment for visibility in GitLab’s Environments dashboard.

Error Handling

• Validate JWTs and fail fast if missing or Vault calls fail.

• Suppress errors (2>/dev/null) to avoid leaking sensitive info.

Manual CI_JOB_JWT Best Practices

Minimize Dependencies

• Use a lightweight image (e.g., alpine) and cache vault in a custom runner image:

• Cache dependencies in a custom runner image to speed up pipelines:

FROM alpine:latest
RUN apk add --no-cache vault

Secure JWT handling

• Never log (or echo) $CI_JOB_JWT - it’s sensitive and could be exploited if exposed.

• Pass it directly to Vault without intermediate storage.

• If avoiding the Vault CLI, use curl for portability

export VAULT_TOKEN=$(curl -s --request POST --data "{\"jwt\": \"$CI_JOB_JWT\", \"role\": \"project-42-role\"}" $VAULT_ADDR/v1/auth/jwt/login | jq -r .auth.client_token)

Manual id_tokens (without secrets) Best Practices

Audience Specificity & JWT Validation

• Set aud in id_tokens to match Vault’s expected audience.

• Set the aud in id_tokens to match your Vault server’s URL exactly, ensuring the JWT is purpose-specific.

• Check $VAULT_ID_TOKEN presence in before_script.

Secrets

• Use fully qualified paths (e.g., secret/project-42/api/api_key) in secrets to avoid ambiguity with Vault mounts.

• Only define secrets you need in the job—don’t pull unnecessary ones.

Manual Flexibility

Use for custom logic (e.g., dynamic roles):

export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role="${CI_PROJECT_ID}-role" jwt=$VAULT_ID_TOKEN)

Modern secrets Best Practices

• Use exact secret paths (e.g., secret/project-42/api/api_key).

• Fetch only required secrets per job.

Don't forget also Vault Best Practices

Use Short-Lived Tokens

• Set token_ttl (e.g., 5m) and token_max_ttl (e.g., 10m) to limit exposure.

• Grant minimal capabilities (e.g., read) to specific paths.

• Enable audit logs in Vault to monitor access.

Why These Practices Matter

• Security: Short TTLs, bound claims, and protected variables minimize risks of token or secret exposure.

• Reliability: Error handling and validation prevent silent failures.

• Maintainability: Audit logs and environment tracking simplify debugging and compliance.

---

Comparison of Methods

---

Testing and Validation

• Pipeline Logs: Look for “API Key retrieved successfully” and ensure no sensitive data (e.g., $VAULT_TOKEN) leaks.

• Vault Audit: Verify in /var/log/vault_audit.log that only authorized roles access secrets.

• Debugging:

  • Manual methods: Add vault token lookup to inspect token capabilities.

  • Modern method: Check $API_KEY presence.

• Dry Run: Test on a non-protected branch to confirm rules block execution.

---

Troubleshooting

• "invalid role": Ensure role name matches (project-42-role) and bound_claims align with pipeline metadata.

• "x509: certificate signed by unknown authority": Set VAULT_CACERT or use VAULT_SKIP_VERIFY=true (testing only).

• "Secret not found": Verify path (secret/project-42/api vs. secret/data/project-42/api) and KV version.

---

Conclusion

Whether you’re stuck on GitLab 13.4 with CI_JOB_JWT, need manual control with id_tokens, or prefer the simplicity of secrets, this guide provides a secure, practical way to integrate GitLab pipelines with Vault. Apply the best practices to minimize risks and streamline your workflow. For self-signed certificates or custom setups, adjust the TLS handling as needed.

---

Need Expert Help with HashiCorp Vault?

🚀 Struggling with Vault integration, security, or scaling? Whether you're using open-source Vault, Vault Enterprise, or Vault Cloud, I can help optimize your setup, enhance security, and streamline your GitLab CI/CD workflows.

Services I offer:

✅ Secure and scalable Vault deployments ✅ CI/CD pipeline optimization with Vault integration ✅ Vault Enterprise and multi-cloud implementations ✅ Security best practices and compliance

📩 Let’s connect! Reach out on LinkedIn or schedule a free consultation and take your Vault setup to the next level.

---

Happy deploying! 🚀