Solving Kubernetes Scheduling Challenges with Taints and Tolerations

Imagine you've got a bunch of servers in your Kubernetes cluster, some powerful, some with specialized hardware like GPUs, and others just your average machines. Now, you've got these applications, or pods as Kubernetes calls them, that need to run somewhere. Some pods are picky, they need that GPU, while others are happy anywhere. How do you make sure the right pods land on the right servers? That's where Taints and Tolerations come in. They're like a matchmaking system for your pods and nodes.

Taints are basically labels you put on a node that say "Hey, not everyone is welcome here!" Maybe it's got a GPU, so you taint it to indicate that only pods specifically requesting a GPU can run there. Tolerations, on the other hand, are like a pod saying "I can handle that taint, let me in!" It's a way for a pod to declare its compatibility with a tainted node.

Without this system, your GPU-hungry pods might end up on regular nodes, struggling to perform, while your less demanding pods hog the specialized hardware. Taints and tolerations bring order to this chaos, ensuring that your resources are used efficiently and your applications run smoothly. In this post, we'll dive deeper into how taints and tolerations work, exploring different types and use cases, so you can master this essential Kubernetes concept.

Let's see how this plays out in a real Kubernetes cluster. The screenshot shows a few key commands in action. First, we check the status of our nodes with kubectl get nodes. We've got a control-plane node and a couple of worker nodes, all happily in a Ready state. Next, we take a closer look at one of the worker nodes, cka-cluster-worker, using kubectl describe node and filtering the output with grep -i taints. As you can see, initially, there are no taints on this node , it's open to all pods.

Now, let's say this worker node has a GPU, and we want to reserve it for pods that specifically need it. We use the kubectl taint command: kubectl taint nodes cka-cluster-worker gpu=true:NoSchedule. This adds a taint to the node cka-cluster-worker. After running this command, Kubernetes confirms that the node has been tainted. Now, any pod that needs to use this GPU will need a corresponding toleration in its configuration.

Now, if we run kubectl describe nodes cka-cluster-worker | grep -i taints again, we can see the taint we just applied: gpu=true:NoSchedule. This confirms that the node is now marked, preventing pods without a specific toleration from being scheduled there. This is how you can verify that your taint has been correctly applied to the node. It's a good practice to double-check after applying taints to ensure your configuration is as intended. This sets the stage for configuring our pods to tolerate this taint, which we'll explore in a later section.

We'll taint another worker node, cka-cluster-worker2, but this time with gpu=false . First, we'll check the status of our nodes using kubectl get nodes and see that we have a control-plane node and two worker nodes. Then, we'll inspect our second worker node, cka-cluster-worker2, for any existing taints (using kubectl describe nodes cka-cluster-worker2 | grep -i taints). As you can see, it's currently taint-free. Now, we'll add a taint to this worker node, specifying that it doesn't have a GPU. We'll use the command kubectl taint nodes cka-cluster-worker2 gpu=false:NoSchedule. This means that pods without a corresponding toleration (saying they can handle the absence of a GPU) won't be scheduled here.

Let's verify that the taint has been applied correctly, we can run the command kubectl describe nodes cka-cluster-worker2 | grep -i taints

Now, let's see what happens when a pod doesn't have the necessary tolerations. We'll create a simple pod running nginx, nothing fancy. The YAML definition for this pod is shown below. Take a close look there's no tolerations section. This means the pod is currently unaware of, and unable to tolerate, any taints on our nodes. It's like trying to enter a members-only club without a membership card.

We'll apply this YAML file to our cluster using kubectl apply -f pod-without-toleration.yaml. Now, remember that we've tainted cka-cluster-worker2 with gpu=false:NoSchedule. So, what do you think will happen to our newly created pod? Let's check its status with kubectl get pods. If you guessed that it would be stuck in a Pending state, you're absolutely right! The pod can't be scheduled because it doesn't have a toleration that matches the taint on the worker nodes. It's effectively blocked from running. This highlights the importance of configuring tolerations when you have tainted nodes in your cluster.

So, how do we get our nginx pod running on the tainted node? We give it a toleration, of course! Below is a slightly modified YAML file. The key difference is the addition of the tolerations section. This section specifies that the pod can tolerate a taint with key: "gpu", operator: "Equal", value: "true", and effect: "NoSchedule". This directly addresses the taint we applied earlier to cka-cluster-worker, effectively giving this pod permission to run on that node.

This configuration tells Kubernetes, "Hey, even though this node is tainted, this pod is okay with it." It's like giving the pod a special pass to bypass the restriction. We'll apply this new YAML file and see if it makes a difference. Let's deploy the pod with the toleration using kubectl apply -f ./cka-prep/pod-with-toleration.yaml. Now, if we check the status of our pods with kubectl get pods, we see the difference! The pod-with-toleration is now happily in a Running state, while the pod-without-toleration remains stuck in Pending. This proves that the toleration we added allowed the pod to be scheduled onto the tainted node. It's like our pod presented the correct credentials and was granted access! This demonstrates how tolerations give you fine-grained control over where your pods can run, even on nodes with specific restrictions.

Okay, let's switch gears and look at how taints affect pods that are already running on a node. This is a crucial aspect of using taints and tolerations in a live environment. We'll use our control-plane node for this example. First, let's get a quick overview of our nodes with kubectl get nodes. As you can see, we have our control-plane and worker nodes ready to go. Now, control-plane nodes typically have a default taint that prevents regular workloads from being scheduled there. Let's check the taints on our control-plane node using kubectl describe nodes cka-cluster-control-plane | grep -i taints. The output shows the taint: node-role.kubernetes.io/control-plane:NoSchedule.

For this experiment, we're going to remove this taint. We can do this with the command: kubectl taint nodes cka-cluster-control-plane node-role.kubernetes.io/control-plane:NoSchedule-. The - at the end signifies that we're removing the taint. . This opens up the control-plane node for general scheduling, at least temporarily.

Now that the control-plane node is taint-free, we can schedule a pod there without needing any tolerations. This sets the stage for our next step, where we'll deploy a pod without toleration to see what happens.

Alright, with the taint removed from our control-plane node, let's deploy a pod. We'll use a simple Redis pod for this example. The YAML definition is shown below, note the absence of any tolerations. Since the control-plane node is currently untainted, this pod should be scheduled without any issues.

After applying the redis-pod.yaml file using kubectl apply -f ./cka-prep/redis-pod.yaml, a quick check with kubectl get pods and kubectl get pods -o wide shows our redis-pod running successfully on the control-plane node. This is a stark contrast to our earlier experience with pod-without-toleration, which remained stubbornly in a Pending state because all nodes were tainted and it lacked the necessary tolerations (as shown in the image below). Now, with the control-plane taint temporarily removed, the scheduler had no problem placing the redis-pod and pod-without-toleration in the control plane node. This highlights the crucial role of taints in controlling pod placement and the importance of matching tolerations to avoid scheduling issues.

Now for the final act – let's re-apply the taint to the control-plane node. We use the command: kubectl taint nodes cka-cluster-control-plane node-role.kubernetes.io/control-plane:NoSchedule. This puts the restriction back in place.

The redis-pod and pod-without-toleration, which was already running on the control-plane node, remains unaffected. This is the key takeaway: the NoSchedule taint prevents new pods from being scheduled, but it doesn't evict existing ones. This behavior is crucial for maintaining the stability of your cluster during taint management. You wouldn't want to accidentally disrupt running applications just by applying a taint. This experiment demonstrates the practical implications of taints and tolerations, showing how they can be used to manage resource allocation and maintain the health of your Kubernetes cluster without causing unnecessary downtime. Understanding these nuances is essential for effectively managing your Kubernetes environment.