Control Issues: From Policy to Practice

You can get a lot done in Kubernetes just by blocking bad stuff at admission time. That’s where we left things in Part 2. We installed Kyverno, wrote policies, and saw workloads getting stopped before they cause trouble. We also saw things like mutations and generating resources on the fly. It was all fairly straightforward.

But how can we take Kyverno to the next level without just writing a couple of ClusterPolicy YAMLs and calling it a day? Policies don’t live in a vacuum. They need testing, exceptions, tuning, and visibility into what’s actually happening in your cluster. And the cool thing is, all of that is built into Kyverno or available in one of its side projects.

In Part 3 of this series, we’re moving past “first policy” territory and into operations. We’ll cover:

• Matching what gets blocked
• Testing policies before they hit the cluster with kyverno test
• Making exceptions without throwing away your guardrails
• Borrowing from the upstream policy library for quick wins like securityContext hardening and volume restrictions
• Observing policy activity with Prometheus metrics and Policy Reporter dashboards

The goal here isn’t to write more YAML. It’s to build a feedback loop where your policies get better, your exceptions are targeted, and you can actually prove the impact of your enforcement. As usual, let's get to it.

---

Match, Exclude, and the Patterns That Follow

Before we pile on tests, exceptions, and dashboards, let’s be crystal clear on how a rule decides it applies and what happens next. Most “why did this block?” mysteries boil down to match logic or pattern evaluation, simple as that..

Rule Evaluation Order

For each incoming request, Kyverno runs through this sequence:

1. Match – does the resource match match.resources? If not, skip.
2. Exclude – does it also match exclude.resources? If yes, skip.
3. Preconditions – optional extra checks (e.g. JMESPath). If false, skip.
4. Action – run the rule (validate, mutate, generate, or verify).

Note: mutations always happen before validations.

If you’re confused about match vs exclude, you’re not alone. The docs aren’t explicit about ordering, but the source makes it clear: match first, exclude second. Of course that makes sense as optimal, but still.

AND vs OR Logic

• Inside a single resources block, fields are ANDed. All must match.
• Use any: to OR multiple resources blocks.

# Pods in prod namespace AND labeled app=backend
match:
  resources:
    kinds: ["Pod"]
    namespaces: ["prod"]
    selector:
      matchLabels:
        app: backend

# Pods in prod namespace OR labeled app=backend
match:
  any:
    - resources:
        kinds: ["Pod"]
        namespaces: ["prod"]
    - resources:
        kinds: ["Pod"]
        selector:
          matchLabels:
            app: backend

Match Quick Reference

To use:

• resources: select by names, namespaces, kinds, operations, labels, annotations, and namespace selectors.
• subjects: select users, groups, and service accounts.
• roles: select namespaced roles.
• clusterRoles: select cluster-wide roles.

Preconditions

If match/exclude got you in the door, preconditions let you add “only if…” filters. They run after match/exclude but before action.

# Only when hostNetwork=true
preconditions:
  all:
    - key: "{{ request.object.spec.hostNetwork }}"
      operator: Equals
      value: true

Great for scoping rules by field presence, among other things.

Patterns: The Real Work

Once a rule applies, Kyverno still needs to know what inside the YAML you care about. That’s where pattern (or anyPattern) comes in.

Patterns are structural YAML matches, not regexes. You describe the shape/values you expect, and Kyverno checks them.

• pattern — all conditions must be satisfied.
• anyPattern — resource passes if it matches any of the listed patterns.

validate:
  pattern:
    spec:
      securityContext:
        runAsNonRoot: true

validate:
  anyPattern:
    - spec:
        securityContext:
          runAsUser: 1000
    - spec:
        securityContext:
          runAsNonRoot: true

This is just scratching the surface, but it gives a basic overview to use Kyverno effectively.

A Note on CEL

Patterns today are Kyverno’s own YAML-driven DSL with some JMESPath helpers. CEL support is coming (and will eventually unify expression logic across Kubernetes), but for now: stick with patterns.

---

Testing Policies with kyverno test

Before you unleash a new policy on your cluster, it’s worth testing it locally. That’s what kyverno test is for: simulating policy evaluations against sample resources, without creating or blocking anything in Kubernetes.

Unlike kyverno apply, which is handy for quick checks, kyverno test is built for programmatic, repeatable testing. It evaluates match criteria (kinds, namespaces, labels, annotations) exactly like the admission controller would, so you can see which rules apply and which get skipped.

Example Policy

Here’s our block-hostpath.yaml validating policy from Part 2:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-hostpath
spec:
  validationFailureAction: Enforce
  rules:
    - name: disallow-hostpath
      match:
        resources:
          kinds:
            - Pod
          selector:
            matchLabels:
              app: kyverno-demo
      validate:
        message: "hostPath volumes are not allowed."
        pattern:
          spec:
            volumes:
              - =(hostPath): "absent"

Notice the selector, which means this rule only applies to Pods labeled app=kyverno-demo. If your resource doesn’t match that, the test will skip it.

Setting Up a Test Directory

Let's create a place to store our tests. This will make it more manageable as you would expect.

mkdir kyverno-tests
cd kyverno-tests
cp /path/to/block-hostpath.yaml .

Create a Passing and Failing Pod

Bad Pod (should fail):

apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
  labels:
    app: kyverno-demo
spec:
  containers:
    - name: nginx
      image: nginx
  volumes:
    - name: root-mount
      hostPath:
        path: /

Good Pod (should pass):

apiVersion: v1
kind: Pod
metadata:
  name: good-pod
  labels:
    app: kyverno-demo
spec:
  containers:
    - name: nginx
      image: nginx

Create a Test

Now let’s create a test manifest to check both pods against the policy.

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: kyverno-test
policies:
  - block-hostpath.yaml
resources:
  - bad-pod.yaml
  - good-pod.yaml
results:
- policy: block-hostpath
  rule: disallow-hostpath
  result: pass

This tells the Kyverno CLI three things:

• policies: which policy files to load (block-hostpath.yaml)
• resources: which resource manifests to run those policies against (bad-pod.yaml, good-pod.yaml)
• results: what you expect to happen. Here, the disallow-hostpath rule should pass for the given resource.

Running the Test

Save the manifest as kyverno-test.yaml and run:

kyverno test .

Example output:

│ ID │ POLICY         │ RULE              │ RESOURCE                │ RESULT │ REASON              │
│────│────────────────│───────────────────│─────────────────────────│────────│─────────────────────│
│ 1  │ block-hostpath │ disallow-hostpath │ v1/Pod/default/good-pod │ Fail   │ Want pass, got fail │
│ 2  │ block-hostpath │ disallow-hostpath │ v1/Pod/default/bad-pod  │ Fail   │ Want pass, got fail │

Test Summary: 0 tests passed and 2 tests failed
Error: 2 tests failed

The pods respect the policy when applied to a live cluster, but the test fails. Why? Because of how we wrote the pattern:

pattern:
  spec:
    volumes:
      - =(hostPath): "absent"

If the volumes block is completely missing, the test still fails. To handle this, we need anyPattern.

Fixing with anyPattern

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-hostpath-updated
spec:
  validationFailureAction: Enforce
  rules:
    - name: disallow-hostpath
      match:
        resources:
          kinds: ["Pod"]
          selector:
            matchLabels:
              app: kyverno-demo
      validate:
        message: "hostPath volumes are not allowed."
        anyPattern:
          - spec:
              =(volumes): "absent"
          - spec:
              volumes:
                - =(hostPath): "absent"

Update the test to use this policy:

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: kyverno-test
policies:
  - block-hostpath-updated.yaml
resources:
  - bad-pod.yaml
  - good-pod.yaml
results:
- policy: block-hostpath-updated
  rule: disallow-hostpath
  result: pass

Run again:

│ ID │ POLICY                 │ RULE              │ RESOURCE                │ RESULT │ REASON │
│────│────────────────────────│───────────────────│─────────────────────────│────────│────────│
│ 1  │ block-hostpath-updated │ disallow-hostpath │ v1/Pod/default/good-pod │ Pass   │ Ok     │
│ 2  │ block-hostpath-updated │ disallow-hostpath │ v1/Pod/default/bad-pod  │ Fail   │ Want pass, got fail │

Test Summary: 1 tests passed and 1 tests failed
Error: 1 tests failed

Perfect! The good pod now passes while the bad pod still fails.

Next Steps

There’s much more you can do with tests, including variables, JSON patches, and negative cases. If you want to level up further, check out Chainsaw, a more advanced testing project for Kyverno.

---

Exceptions (Two Practical Paths)

You’ll need exceptions. The trick is making them surgical, not blanket “turn it all off.” Here are two clean approaches.

Option 1 — Use the exclude Block (Fast and Built-In)

Keep a single global policy and carve out narrowly with exclude. Four common exclusion types:

Namespace carve-out:

exclude:
  resources:
    namespaces: ["legacy-systems", "migration", "bleeding-edge"]

Predictable, but blunt. Whole namespaces get a hall pass.

Label-based carve-out:

exclude:
  resources:
    selector:
      matchLabels:
        kyverno-exempt: "true"

Tactical: mark a pod with kyverno-exempt=true and it skips evaluation.

Role-based carve-out:

exclude:
  any:
  - clusterRoles:
    - cluster-admin

If cluster-admin creates it, hands off. (Sometimes you need to respect the crown.)

Subject-targeted carve-out:

exclude:
  - subjects:
    - kind: User
      name: CloudSecBurrito

Skip checks for one named user. Minimal and very explicit.

---

Option 2 — PolicyException CRD (Surgical and Auditable)

When you need to exempt one rule of one policy for a specific target, use the PolicyException CRD. It doesn’t touch the original policy, and it’s easy to audit later.

You’ll need to enable it (disabled by default). Example flow:

kubectl create namespace kyverno-exceptions

kubectl -n kyverno patch deploy kyverno-admission-controller --type='json' -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--enablePolicyException=true"},
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--exceptionNamespace=kyverno-exceptions"}
]'

Check which versions are supported:

kubectl get crd policyexceptions.kyverno.io -o jsonpath='{range .spec.versions[*]}{.name}{"\t"}{.served}{"\t"}{.storage}{"\n"}{end}'

If you see v2 true true, your manifest should use apiVersion: kyverno.io/v2.

Example exception — exempt one pod from the block-hostpath rule:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: allow-special-pod
  namespace: kyverno-exceptions
spec:
  exceptions:
    - policyName: block-hostpath
      ruleNames: ["disallow-hostpath"]
  match:
    any:
      - resources:
          kinds: ["Pod"]
          namespaces: ["default"]
          names: ["special-pod"]

Now that pod runs, but the policy stays enforced everywhere else. Surgical, auditable, and controlled.

---

Monitoring Kyverno with Prometheus

You don’t need 12 dashboards to prove policies work. Just scrape Kyverno, run a couple queries, and move on. Prometheus pros can skip ahead.

Basic setup (kube-prometheus-stack)

Deploy Prometheus (yes, Grafana comes along for the ride, we’re ignoring it):

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace monitoring
helm install kube-prom prometheus-community/kube-prometheus-stack -n monitoring

Patch it to NodePort so you can hit it locally:

kubectl patch svc kube-prom-kube-prometheus-prometheus   -n monitoring   -p '{"spec": {"type": "NodePort"}}'
kubectl get svc kube-prom-kube-prometheus-prometheus -n monitoring -o wide

Access Prometheus at something like http://192.168.64.7:31559/. Cool. Now onto the ServiceMonitor.

Configure a ServiceMonitor for Kyverno

Create a ServiceMonitor that matches kube-prometheus-stack’s selector (release: kube-prom):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kyverno-metrics
  namespace: kyverno
  labels:
    release: kube-prom   # IMPORTANT: must match Prometheus CR serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kyverno-admission-controller
  namespaceSelector:
    matchNames:
      - kyverno
  endpoints:
    - targetPort: 8000
      path: /metrics
      interval: 30s

Apply it, then check /targets in Prometheus UI to confirm Kyverno is UP.

A Couple Queries

Failures for Pods in the default namespace over the last 5 minutes (create something broken first so it is guaranteed to show data):

kyverno_policy_results_total{rule_result="fail",resource_kind="Pod",resource_namespace="default"}[5m]

Policy changes (create a new policy, then refresh the query):

kyverno_policy_changes_total

And for a quick heartbeat check, make sure Kyverno’s alive at all:

kyverno_info

That’s it. Prometheus is scraping Kyverno, queries return real numbers, and you didn’t even need to pretend to like dashboards.

---

Policy Reporter – Your Friendly Dashboard

For the finale, let’s spin up a dashboard. No Grafana this time — we’ll use the adjacent project Policy Reporter.

Setup

Deploy with Helm:

helm repo add policy-reporter https://kyverno.github.io/policy-reporter
helm repo update
helm install policy-reporter policy-reporter/policy-reporter --create-namespace -n policy-reporter --set ui.enabled=true

Expose it via NodePort:

kubectl patch -n policy-reporter svc policy-reporter-ui -p '{"spec": {"type": "NodePort"}}'
kubectl get svc -n policy-reporter policy-reporter-ui -o wide

Now hit it in your browser just like Prometheus, e.g. http://192.168.64.7:31864/.

You’ll see a clean UI with policy passes, failures, and violations grouped by rule and namespace. Instant dashboards for your Kyverno policies. Turns out not all dashboards are evil after all.

---

Borrowing from the Upstream Policy Library

Kyverno ships with an extensive policy library you can pull from for quick, high-impact wins. Instead of writing every rule yourself, lean on what’s already been proven. A few highlights worth adopting right away:

• Restrict automounting service account tokens
  Policy link
  Prevents workloads from automatically mounting a service account token unless explicitly allowed. Cuts down on “accidental” privilege handouts.

• Block cluster-admin role bindings
  Policy link
  Stops developers (or attackers) from casually granting themselves cluster-admin. Because least privilege means least.

• Deny role escalation verbs
  Policy link
  Prevents roles from including verbs like escalate or impersonate that let users jump trust boundaries.

• Ban wildcard verbs in roles
  Policy link
  Avoids the dreaded * in RBAC rules. Force teams to think about what permissions they actually need, rather than granting everything by default.

Each of these tackles real-world abuse paths we’ve seen exploited. They’re a fast way to raise the floor on cluster security without spending weeks hand-crafting policies.

---

Wrap Up

We’ve covered a lot of ground in this trilogy.

In Part 1, we got down to the nitty-gritty of raw admission control — even wiring up our own admission controller. We also looked at Pod Security Admission, Kubernetes’ built-in controller that’s… let’s just say, not exactly the sharpest tool in the shed.

In Part 2, we dove into Kyverno itself. We covered the different policy types and ran through concrete, working examples.

This series coincided with my BSides Las Vegas talk (jump to ~17:13 if you want to watch me sweat through slides). I picked Kyverno partly because of that talk, but mostly because I genuinely believe admission control should be mandatory in any Kubernetes cluster.

I haven’t given Gatekeeper or jsPolicy their fair shake yet, but I will. I started here because Kyverno is approachable, all YAML, and still powerful. Between the CLI, policy types, and extra tooling like dashboards, it gives you everything you need to get real work done. You don’t need to gold-plate everything to be effective; Kyverno strikes a good balance between power and usability.

The time you invest in Kyverno will pay off. Big time. And with that, after three posts, one conference talk, and far too much self-flattery, we can finally call it a wrap.