From Static Keys to Dynamic Secrets: A Modern SecOps Blueprint
Atif FarrukhWe’ve all seen it. That one critical file production.tfvars or secrets.env, containing master database password or your cloud provider keys. It’s the skeleton in almost every setup, we convince ourselves that we will rotate it “later”, but we all know that “later” never comes. The single static secret becomes the linchpin of the entire system, a ticking bomb of operational risk.
This is the old way of thinking, that secrets are static strings that are to be managed, protected and manually rotated (which we rarely do). From my experience, this approach is fundamentally broken. It’s risky and fail at scale.
A modern Secrets Operations (SecOps) model is built on a foundational principle: there should be no long-lived secrets. Credentials should be ephemeral, generated on-demand and for specific task and time. This principle shifts our focus from protecting the secret to securing and auditing the access to that secret
This is the blueprint for building such a system using HashiCorp Vault and AWS IAM, turning a major security liability into an automated, auditable strength.
The Architecture: The Dynamic Secrets Flywheel
The goal is to eliminate the need for an application to posses a permanent database credential. Instead, we create a system where the application will request temporary credentials from HashiCorp Vault just-in-time.
+-----------------+ +-------+ +-----------------+ +------------+
| App (EC2/EKS) | | AWS | | Vault | | Database |
+-----------------+ +-------+ +-----------------+ +------------+
| | | |
|--- 1. Request IAM Role Identity ------>| | |
| | | |
|<--- IAM Identity Document -------------| | |
| | | |
|--- 2. Present Trusted Identity ------->| | |
| | | |
| |<- 3a. Verify Identity w/ AWS -| |
| | (Checks Policy) | |
| | |--- 4. Create Temp DB User -->|
| | | |
| | |<---- New Credentials --------|
| | | |
|<-- 5. Return Temporary Username/Password ------------------------------| |
| | | |
|-------------------------- 6. Connect to DB w/ Temp Credentials ------------------------------> |
| | | |
| | | |
| | |- 7. Revoke Credential ----->|
| | | (when TTL expires) |
| | | |
In this model, even if a credential were to leak, its lifespan is of the credentials will be dependant on TTL, not years, dramatically reducing the window of opportunity for an attacker.
Implementation Details: AWS Secret Manager
Let's walk through a production-grade implementation using AWS Secret Manage to generate dynamic credentials for an Amazon RDS (PostgreSQL) database.
- The "Secret Zero" Bootstrap: Authenticating to AWS
The first problem is how our application proves its identity to AWS. We solve this with the AWS Auth Method.
Terraform to configure the AWS Auth Method:
# IAM Policy to allow microservices to read their specific secrets
resource "aws_iam_policy" "microservice_secret_reader_policy" {
name = "MicroserviceSecretReaderPolicy"
description = "Allows microservices to read their specific database secrets"
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
Resource = [for s in aws_secretsmanager_secret.microservice_db_secret : s.arn]
}
]
})
}
# IAM Role for each microservice using IRSA
resource "aws_iam_role" "microservice_irsa_role" {
for_each = local.microservices_eks_config
name = "${each.key}-irsa-role" # e.g., "user_service-irsa-role"
# This trust policy is crucial for IRSA
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Principal = {
Federated = data.aws_iam_openid_connect_provider.main.arn
},
Action = "sts:AssumeRoleWithWebIdentity",
Condition = {
StringEquals = {
"${replace(data.aws_iam_openid_connect_provider.main.url, "https://", "")}:sub" = "system:serviceaccount:${each.value.k8s_namespace}:${each.value.k8s_service_account_name}"
"${replace(data.aws_iam_openid_connect_provider.main.url, "https://", "")}:aud" = "sts.amazonaws.com"
}
}
}
]
})
# Attach the Secrets Manager policy to each microservice's IRSA role
resource "aws_iam_role_policy_attachment" "microservice_secret_attachment" {
for_each = local.microservices_eks_config
policy_arn = aws_iam_policy.microservice_secret_reader_policy.arn
role = aws_iam_role.microservice_irsa_role[each.key].name
}
# Kubernetes Service Account for each microservice
resource "kubernetes_service_account_v1" "microservice_sa" {
for_each = local.microservices_eks_config
metadata {
name = each.value.k8s_service_account_name
namespace = each.value.k8s_namespace
annotations = {
# This annotation links the K8s Service Account to the AWS IAM Role
"eks.amazonaws.com/role-arn" = aws_iam_role.microservice_irsa_role[each.key].arn
}
}
}
This solves the "secret zero" problem. Our application doesn't need a AWS secrets to start; its inherent IAM identity is its key to the front door.
- Configure the Dynamic Secret Engine
Next, we configure AWS Secret Manager to connect to our RDS database.
Terraform to configure the Database Secrets Engine:
# Define the Lambda role for rotation
resource "aws_iam_role" "secret_rotation_lambda_role" {
name = "SecretRotationLambdaRole"
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Principal = {
Service = "lambda.amazonaws.com"
},
Action = "sts:AssumeRole"
}
]
})
}
resource "aws_iam_policy" "secret_rotation_lambda_policy" {
name = "SecretRotationLambdaPolicy"
description = "Policy for Secrets Manager rotation Lambda"
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage",
"secretsmanager:TestSecret"
],
Resource = [for s in aws_secretsmanager_secret.microservice_db_secret : s.arn]
},
{
Effect = "Allow",
Action = [
"rds:DescribeDBInstances",
"rds:ModifyDBInstance" # This is for master user rotation. For dedicated users, it's just about connecting.
],
Resource = aws_db_instance.main_postgres_instance.arn
},
{
Effect = "Allow",
Action = "lambda:InvokeFunction",
Resource = "*" # restrict this to lambda ARN
},
{
Effect = "Allow",
Action = "logs:CreateLogGroup",
Resource = "arn:aws:logs:*:*:*"
},
{
Effect = "Allow",
Action = [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
Resource = "arn:aws:logs:*:*:log-group:/aws/lambda/*:*"
}
]
})
}
resource "aws_iam_role_policy_attachment" "secret_rotation_lambda_policy_attach" {
policy_arn = aws_iam_policy.secret_rotation_lambda_policy.arn
role = aws_iam_role.secret_rotation_lambda_role.name
}
- Tying It Together with a Application Policy
Finally lets connect everything together.
# IAM Policy to allow microservices to read their specific secrets
resource "aws_iam_policy" "microservice_secret_reader_policy" {
name = "MicroserviceSecretReaderPolicy"
description = "Allows microservices to read their specific database secrets"
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
Resource = [for s in aws_secretsmanager_secret.microservice_db_secret : s.arn]
}
]
})
}
# IAM Role for a generic microservice
# For ECS tasks, this would be an `aws_iam_role` and `aws_iam_role_policy_attachment`
# For EKS, you'd use IRSA (IAM Roles for Service Accounts) which involves
# `aws_iam_role` and `aws_iam_policy_attachment` linked to an EKS Service Account.
# For simplicity, let's create a generic role that assumes EC2 or ECS task roles.
resource "aws_iam_role" "microservice_role" {
name = "MicroserviceExecutionRole"
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Principal = {
Service = "ecs-tasks.amazonaws.com" # For ECS
# "ec2.amazonaws.com" # If running directly on EC2
},
Action = "sts:AssumeRole"
}
]
})
tags = {
Name = "MicroserviceExecutionRole"
}
}
resource "aws_iam_role_policy_attachment" "microservice_secret_attachment" {
policy_arn = aws_iam_policy.microservice_secret_reader_policy.arn
role = aws_iam_role.microservice_role.name
}
With these pieces in place, an application can now fetch its credentials programmatically, without ever seeing a static password.
Pitfalls & Optimizations
IAM Roles for Service Accounts (IRSA) Misconfiguration: Incorrectly configured IAM trust policies or Kubernetes Service Account annotations. If the
eks.amazonaws.com/role-arnannotation is missing, wrong, or the IAM role's trust policy doesn't correctly match the OIDC provider and service account, your pods won't be able to assume the role and will fail to retrieve secrets.Lease Management Complexity: Your application is now responsible for understanding leases, renewing them before they expire, and requesting a new secret if the old one is revoked. This adds complexity to your application code.
Unlocked: The Final Takeaway
Modern Secrets Operations is a paradigm shift.
You stop managing secrets; you start managing access. The focus moves from protecting a static string to defining and auditing the policies that govern who (or what) can request access.
Credentials become ephemeral and disposable. By making secrets short-lived, you drastically reduce the value of a compromised credential.
Security becomes automated and auditable. The entire process is defined in code and every action is logged, providing a clear, verifiable trail for compliance and incident response.
This framework requires an upfront investment in architecture and tooling, but it is the only way to truly secure a dynamic, cloud-native environment at scale. It replaces the anxiety of manual secret rotation with the confidence of automated, just-in-time security.
Implementing a dynamic secrets infrastructure is a foundational step in building a mature, secure, and compliant cloud platform.
If your organization is ready to move beyond static secrets and build a true SecOps foundation, I can help you design and implement the solution.
Email me for a detailed architectural discussion: atif@devopsunlocked.dev
Explore my projects and connect on Upwork: https://www.upwork.com/freelancers/~0118e01df30d0a918e
Subscribe to my newsletter
Read articles from Atif Farrukh directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by