Introduction

Kubernetes scheduling is a complex orchestration process that determines where pods should run across your cluster. While node selectors and affinity rules help you specify where pods should run, taints and tolerations work in the opposite direction - they allow nodes to repel pods that shouldn't run on them.

Think of taints as a "keep out" sign on nodes, and tolerations as special passes that allow certain pods to ignore these signs. This mechanism is crucial for maintaining cluster hygiene, ensuring workload isolation, and managing specialized hardware resources.

Understanding Taints and Tolerations

What are Taints?

Taints are key-value pairs applied to nodes that act as repellents for pods. When a node is tainted, it will not accept any pods that do not have a matching toleration. Taints consist of three components:

Key: A string that identifies the taint
Value: An optional string value associated with the key
Effect: Defines what happens to pods that cannot tolerate the taint

What are Tolerations?

Tolerations are applied to pods and allow them to schedule onto nodes with matching taints. A toleration "tolerates" a taint, meaning it can coexist with it. Tolerations have similar components to taints but include additional matching operators.

Taint Effects

There are three types of taint effects:

NoSchedule: Prevents new pods from being scheduled on the node
PreferNoSchedule: Kubernetes will try to avoid placing pods on the node, but it's not guaranteed
NoExecute: Evicts existing pods from the node and prevents new ones from being scheduled

Hands-On Lab Setup

Let's set up a practical lab environment using a multi-node Kubernetes cluster for our examples. You can use platforms like KodeKloud, Minikube, or KillerKoda to create and manage your cluster.

Prerequisites

# Verify your cluster has multiple nodes
kubectl get nodes

# Check current node status
kubectl describe nodes

Lab Environment Preparation

# Create a namespace for our experiments
kubectl create namespace taint-demo

# Set context to our demo namespace
kubectl config set-context --current --namespace=taint-demo

Hands-On Example 1: Basic Taint and Toleration

Step 1: Apply a Taint to a Node

# List available nodes
kubectl get nodes

# NAME           STATUS   ROLES           AGE    VERSION
# controlplane   Ready    control-plane   112m   v1.33.0
# node01         Ready    <none>          111m   v1.33.0
# node02         Ready    <none>          111m   v1.33.0

# Apply a taint to a specific node (replace 'node01' with your actual node name)
kubectl taint nodes node01 environment=production:NoSchedule
# node/node01 tainted

# Verify the taint was applied
kubectl describe node node01 | grep -i taint
# Taints:             environment=production:NoSchedule

Step 2: Deploy a Pod Without Toleration

Create a simple pod that doesn't have any tolerations:

# basic-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: basic-pod
  namespace: taint-demo
spec:
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80

# Apply the pod
kubectl apply -f basic-pod.yaml
# namespace/taint-demo created

# Check pod status - it should be pending if scheduled on tainted node
kubectl get pods -o wide
# NAME        READY   STATUS    RESTARTS   AGE   IP           NODE
# basic-pod   1/1     Running   0          25s   172.17.2.2   node02
# In my case, it is scheduled on node02

# Check events to see scheduling issues
kubectl describe pod basic-pod

Step 3: Create a Pod with Matching Toleration

# tolerant-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: tolerant-pod
  namespace: taint-demo
spec:
  tolerations:
  - key: "environment"
    operator: "Equal"
    value: "production"
    effect: "NoSchedule"
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80

# Apply the tolerant pod
kubectl apply -f tolerant-pod.yaml
# pod/tolerant-pod created

kubectl get pods -o wide
# NAME           READY   STATUS    RESTARTS   AGE     IP           NODE
# basic-pod      1/1     Running   0          5m59s   172.17.2.2   node02
# tolerant-pod   1/1     Running   0          51s     172.17.1.2   node01

💡✨ Quick Analogy

Think of tolerations like a visitor carrying a gas mask (toleration) — it’s only necessary if entering a smoky room (tainted node). But if the room has clear air (no taint), the gas mask doesn’t hurt — and the visitor can still enter.

Hands-On Example 2: NoExecute Effect

The NoExecute effect is particularly powerful as it can evict running pods.

Step 0: Remove the Existing Node Taint

# The command to untaint a node is similar to the taint command 
# but with a hyphen (-) at the end of the taint specification.
kubectl taint nodes node01 environment=production:NoSchedule-
# node/node01 untainted

Step 1: Create Pods on All Nodes

# Create a deployment to spread pods across nodes
kubectl create deployment test-deployment --image=nginx --replicas=6
#deployment.apps/test-deployment created

kubectl get pods -o wide --selector app=test-deployment
# NAME                               READY   STATUS    RESTARTS   AGE  IP           NODE
# test-deployment-6bb5b6b89f-cjkkj   1/1     Running   0          8s   172.17.2.4   node02
# test-deployment-6bb5b6b89f-cpmz8   1/1     Running   0          8s   172.17.1.3   node01
# test-deployment-6bb5b6b89f-sl2d5   1/1     Running   0          8s   172.17.1.5   node01
# test-deployment-6bb5b6b89f-w95t9   1/1     Running   0          8s   172.17.2.3   node02
# test-deployment-6bb5b6b89f-wfpll   1/1     Running   0          8s   172.17.2.5   node02
# test-deployment-6bb5b6b89f-xjkkj   1/1     Running   0          8s   172.17.1.4   node01

Step 2: Apply NoExecute Taint

# Apply NoExecute taint to a node with running pods
kubectl taint nodes node01 critical=true:NoExecute
# node/node01 tainted

# Watch pods get evicted
kubectl get pods -o wide --selector app=test-deployment -w

# NAME                               READY   STATUS    RESTARTS   AGE  IP           NODE
# test-deployment-6bb5b6b89f-c582s   1/1     Running   0          8s   172.17.2.7   node02
# test-deployment-6bb5b6b89f-cjkkj   1/1     Running   0          1m   172.17.2.4   node02
# test-deployment-6bb5b6b89f-n5kpg   1/1     Running   0          8s   172.17.2.8   node02
# test-deployment-6bb5b6b89f-t6n2h   1/1     Running   0          8s   172.17.2.6   node02
# test-deployment-6bb5b6b89f-w95t9   1/1     Running   0          1m   172.17.2.3   node02
# test-deployment-6bb5b6b89f-wfpll   1/1     Running   0          1m   172.17.2.5   node02

Step 3: Deploy Pod with NoExecute Toleration

# noexecute-tolerant-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: noexecute-tolerant-pod
  namespace: taint-demo
spec:
  tolerations:
  - key: "critical"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
    tolerationSeconds: 300
  containers:
  - name: nginx
    image: nginx:latest

kubectl apply -f noexecute-tolerant-pod.yaml
# pod/noexecute-tolerant-pod created

kubectl get pods -o wide
# NAME                               READY   STATUS    RESTARTS   AGE     IP           NODE     NOMINATED NODE   READINESS GATES
# noexecute-tolerant-pod             1/1     Running   0          2m2s    172.17.1.6   node01   <none>           <none>

Note the tolerationSeconds field - this pod will be evicted after 300 seconds even with the toleration.

Hands-On Example 3: Multiple Taints and Tolerations

Step 1: Apply Multiple Taints

# Apply multiple taints to a node
kubectl taint nodes node02 environment=staging:NoSchedule
kubectl taint nodes node02 hardware=gpu:NoSchedule
kubectl taint nodes node02 team=ml:PreferNoSchedule

# multi-tolerant-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: multi-tolerant-pod
  namespace: taint-demo
spec:
  tolerations:
  - key: "environment"
    operator: "Equal"
    value: "staging"
    effect: "NoSchedule"
  - key: "hardware"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  - key: "team"
    operator: "Equal"
    value: "ml"
    effect: "PreferNoSchedule"
  containers:
  - name: tensorflow
    image: tensorflow/tensorflow:latest
    command: ["sleep", "infinity"]
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"

kubectl apply -f multi-tolerant-pod.yaml
kubectl get pods -o wide
# NAME                 READY   STATUS    RESTARTS   AGE   IP           NODE
# multi-tolerant-pod   1/1     Running   0          10s   172.17.2.3   node02

Hands-On Example 4: Real-World Scenario - GPU Nodes

Let's simulate a real-world scenario where we have dedicated GPU nodes.

# Simulate tainting a GPU node
kubectl taint nodes node01 hardware=gpu:NoSchedule
kubectl taint nodes node01 gpu-type=nvidia-v100:NoSchedule

Step 2: Deploy GPU Workload

# gpu-workload.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-workload
  namespace: taint-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpu-workload
  template:
    metadata:
      labels:
        app: gpu-workload
    spec:
      tolerations:
      - key: "hardware"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      - key: "gpu-type"
        operator: "Equal"
        value: "nvidia-v100"
        effect: "NoSchedule"
      containers:
      - name: gpu-app
        image: tensorflow/tensorflow:latest-gpu
        command: ["sleep", "infinity"]
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            memory: "2Gi"
            cpu: "1000m"

kubectl apply -f gpu-workload.yaml
kubectl get pods -o wide

Hands-On Example 5: Node Maintenance Scenario

Step 1: Drain Node for Maintenance

# Cordon the node first
kubectl cordon node02

# Apply maintenance taint
kubectl taint nodes node02 maintenance=true:NoExecute

# Watch pods get evicted
kubectl get pods -o wide -w

# critical-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: critical-monitoring-pod
  namespace: taint-demo
spec:
  tolerations:
  - key: "maintenance"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
  - key: "node.kubernetes.io/unschedulable"
    operator: "Exists"
    effect: "NoSchedule"
  containers:
  - name: monitoring
    image: prom/node-exporter:latest
    ports:
    - containerPort: 9100

kubectl apply -f critical-pod.yaml
kubectl get pods -o wide

Built-in Taints and Tolerations

Kubernetes automatically applies certain taints and tolerations:

Common Built-in Taints

# Check for built-in taints
kubectl describe nodes | grep -i taint

# Common built-in taints include:
# node.kubernetes.io/not-ready
# node.kubernetes.io/unreachable
# node.kubernetes.io/disk-pressure
# node.kubernetes.io/memory-pressure
# node.kubernetes.io/pid-pressure
# node.kubernetes.io/network-unavailable

DaemonSet Example with Built-in Tolerations

# monitoring-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
  namespace: taint-demo
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      tolerations:
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 300
      - key: "node.kubernetes.io/unreachable"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 300
      - key: "node.kubernetes.io/disk-pressure"
        operator: "Exists"
        effect: "NoSchedule"
      - key: "node.kubernetes.io/memory-pressure"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: monitor
        image: prom/node-exporter:latest
        ports:
        - containerPort: 9100

kubectl apply -f monitoring-daemonset.yaml
kubectl get pods -o wide

Advanced Toleration Patterns

Time-based Tolerations

# time-based-toleration.yaml
apiVersion: v1
kind: Pod
metadata:
  name: time-based-pod
  namespace: taint-demo
spec:
  tolerations:
  - key: "maintenance"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
    tolerationSeconds: 600  # Tolerate for 10 minutes
  containers:
  - name: app
    image: nginx:latest

Conditional Tolerations in Deployments

# conditional-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: conditional-deployment
  namespace: taint-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: conditional-app
  template:
    metadata:
      labels:
        app: conditional-app
    spec:
      tolerations:
      - key: "environment"
        operator: "In"
        values: ["production", "staging"]
        effect: "NoSchedule"
      containers:
      - name: app
        image: nginx:latest

Troubleshooting Common Issues

Issue 1: Pods Stuck in Pending State

# Check for scheduling issues
kubectl describe pod <pod-name>

# Look for taint-related events
kubectl get events --sort-by=.metadata.creationTimestamp

# Check node taints
kubectl describe nodes | grep -A 5 -B 5 -i taint

Issue 2: Pods Being Evicted Unexpectedly

# Check for NoExecute taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Check pod tolerations
kubectl get pod <pod-name> -o yaml | grep -A 10 tolerations

Conclusion

Taints and tolerations are powerful Kubernetes features that provide fine-grained control over pod scheduling. They're essential for:

Resource Management: Dedicating nodes to specific workloads
Maintenance Operations: Safely draining nodes for updates
Quality of Service: Ensuring critical workloads get appropriate resources
Multi-tenancy: Isolating different teams or applications
Hardware Optimization: Directing workloads to appropriate hardware

By mastering taints and tolerations, you can build more resilient, efficient, and well-organized Kubernetes clusters. Remember to use them judiciously - overuse can lead to complex scheduling constraints that are difficult to troubleshoot.

The key is to start simple, document your decisions, and gradually build more sophisticated taint strategies as your cluster and workloads evolve. Always test your taint and toleration configurations in a development environment before applying them to production clusters.

Kubernetes Taints and Tolerations: A Complete Guide with Hands-On Examples

Table of contents