Implementing Cluster level Mutual TLS in two Nginx Pods

Recently I was assigned with the task for establishing the MTLS communication between two pods. In this blog I will share my experience about implementing MTLS. Here I have created two Nginx pods and cert-manager to generate certificates

Mutual TLS

In normal TLS we have server certificate which is verified by client to check authenticity of connection. In Mutual TLS both client and server has their certificates and they exchange them to verify their identities.

Setup

• Requirements

  • Kubernetes Cluster

  • Created Separate namespace

  • Two Nginx Pods

  • Cert Manager Installed on Cluster

To install Cert Manager on Cluster you can refer there official documentation. I am using latest version (v1.17.0). Apply below command to install cert-manager.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.0/cert-manager.yaml

In my case I am using Microk8s to create a cluster and created a separate namespace named cert-tls for implementation.

Implementation

• mTLS ensures both client and server authenticate each other using certificates.

• Client and server certificates are stored in Kubernetes secrets.

• The CA Issuer validates these certificates, enabling a secure, trusted connection.

So now we have cluster and namespace setup with cert manager running, we will now create YAML files to create certificates and keys. We have to create following YAML files:

• Self Signed Issuer

    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: selfsigned-issuer
      namespace: cert-tls
    spec:
      selfSigned: {}
  

  Here we created self signed issuer to issue the certificate it is used as Certificate Authority (CA). Here I created kind as Issuer which is valid only for my namespace to create cluster scope issuer use ClusterIssuer.

• CA Certificate

    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: ca-cert
      namespace: cert-tls
    spec:
      isCA: true
      commonName: "cert-tls-ca"
      secretName: ca-key-pair
      duration: 8760h
      renewBefore: 360h
      issuerRef:
        name: selfsigned-issuer
        kind: Issuer
  

  After this we create the CA Certificate referring the issuer selfsigned-issuer. It is used to sign other certificates. It acts as your private root CA. As we can see it is valid of a year and automatically renews in 15 days.

Here we can see two certs ca.crt and tls.crt both are same as ca.crt is generated as we put isCA: True in YAML File. Use tls.crt + tls.key for signing new certificates and ca.crt to mount into a client pod or app to trust this CA.

•   kubectl get secret ca-key-pair -n cert-tls -o jsonpath='{.data}' | jq 'keys'
    # output 
    [
      "ca.crt",
      "tls.crt",
      "tls.key"
    ]
    # certificates or key can also be retrived using following command
    kubectl get secret my-cert-tls -n default -o jsonpath="{.data.tls\.crt}" | base64 --decode > tls.crt
  

• CA Issuer

    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ca-issuer
      namespace: cert-tls
    spec:
      ca:
        secretName: ca-key-pair
  

  Now CA Issuer is created which is used to sign the certificates using the root CA we just created (ca.crt) and stored in ca-key-pair secret.

• Server and Client Certificates

# Server

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mtls-server-cert
  namespace: cert-tls
spec:
  secretName: mtls-server-secret
  duration: 8760h              
  renewBefore: 720h            
  commonName: server.mtls.local
  dnsNames:
    - server.mtls.local
  issuerRef:
    name: my-ca-issuer         
    kind: Issuer

# Client

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mtls-client-cert
  namespace: cert-tls
spec:
  secretName: mtls-client-secret
  duration: 8760h
  renewBefore: 720h
  commonName: client.mtls.local
  dnsNames:
    - client.mtls.local
  issuerRef:
    name: my-ca-issuer
    kind: Issuer

Here we are creating certificate to load onto the server and client pod. It generates the private key and creates Certificate Signing Request (CSR) for the common name and use my-ca-issuer which is our CA issuer to sign the certificate. It stores the certificate and key in Kubernetes Secret. Again this secret will contain ca.crt as trust chain certificate to verify it and tls.crt and private key for server.mtls.local and client.mtls.local

• Server Deployment and Test Client Pod

# Server Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mtls-server
  namespace: cert-tls
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mtls-server
  template:
    metadata:
      labels:
        app: mtls-server
    spec:
      containers:
        - name: nginx
          image: nginx:1.21-alpine
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: tls
              mountPath: /etc/nginx/certs
              readOnly: true
            - name: config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
      volumes:
        - name: tls
          secret:
            secretName: my-service-tls
        - name: config
          configMap:
            name: mtls-nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mtls-nginx-config
  namespace: cert-tls
data:
  nginx.conf: |
    events {}
    http {
      server {
        listen 8443 ssl;
        ssl_certificate /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;
        ssl_client_certificate /etc/nginx/certs/ca.crt;
        ssl_verify_client on;

        location / {
          return 200 'Hello from secure mTLS NGINX Server!';
        }
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: mtls-server
  namespace: cert-tls
spec:
  selector:
    app: mtls-server
  ports:
    - port: 443
      targetPort: 8443

# Client Pod
apiVersion: v1
kind: Pod
metadata:
  name: mtls-client
  namespace: cert-tls
spec:
  containers:
    - name: curl
      image: curlimages/curl:7.85.0
      command: [ "sleep", "3600" ]
      volumeMounts:
        - name: client-certs
          mountPath: /etc/certs
  volumes:
    - name: client-certs
      secret:
        secretName: my-service-tls

Here I have give server and client deployment and service YAML Files with simple ngnix config file in Config Map. You can modify it.

After than exec into the client pod and run below command to test the connection

curl https://mtls-server.cert-tls.svc --cacert /etc/certs/ca.crt
# output
Hello from a secure mTLS-enabled server!

This gives a high-level overview of the process in a Kubernetes setup for Mutual TLS with cert-manager, using nginx as server and client.