Chapter 2: The kernel locks (Limitations of Spinlocks)

Until now, we have understood how spinlocks work and have seen a practical demo on the Raspberry Pi development board.

Next, we will discuss the limitations of spinlocks.

Although spinlocks prevent preemption on the CPU, they do not prevent the CPU from being interrupted by an interrupt.

Consider a situation where a CPU holds a spinlock to protect a resource, and an interrupt occurs. Since interrupts are high-priority tasks, the CPU will stop its current task and jump to the interrupt handler. Now, imagine that the interrupt handler also attempts to acquire the same spinlock that the current task is holding. This will result in the interrupt handler infinitely spinning, trying to access the lock that is already held by the task it has preempted. This particular situation is called a Deadlock.

To address this situation, the Linux kernel provides the functions spin_lock_irq() and spin_unlock_irq(). These functions, in addition to disabling/enabling preemption, also disable/enable interrupts.

So, let's quickly see the definitions:

void spin_lock_irq(spinlock_t *lock);
void spin_unlock_irq(spinlock_t *lock);

You might think this solution is enough, but it's not. The _irq variant partially fixes the problem. Imagine interrupts are already disabled before your code starts locking. When you call spin_unlock_irq(), it won't just release the lock but also enable interrupts. This can be wrong because spin_unlock_irq() can't tell which interrupts were enabled before locking.

Here's a simple example:

1. Interrupts x and y are disabled before acquiring a spinlock, but z is not.

2. spin_lock_irq() disables all interrupts (x, y, and z) and takes the lock.

3. spin_unlock_irq() enables all interrupts (x, y, and z). However, x and y were originally disabled, leading to an error.

spin_lock_irq() is unsafe if IRQs are already off. When spin_unlock_irq() is called, it will enable all IRQs, even those that were originally disabled. Use spin_lock_irq() only when you know that interrupts are enabled and nothing else has disabled them on the local CPU.

Imagine you save the interrupt status before acquiring the lock and restore it when releasing the lock. This prevents any issues. The kernel provides spin_lock_irqsave & spin_unlock_irqrestore functions for this purpose. They work like spin_lock_irq() & spin_unlock_irq() functions but also save and restore the interrupt status.

These functions are:

void spin_lock_irqsave(spinlock_t *lock, unsigned long flags);
void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags);

Important note:

• spin_lock() and its variants automatically call preempt_disable(), which disables preemption on the local CPU.

• spin_unlock() and its variants call preempt_enable(), which tries to enable preemption.

However, enabling preemption depends on whether other spinlocks are locked, affecting the preemption counter. If preemption is enabled, spin_unlock() may call schedule(), making spin_unlock() a potential preemption point.

Now we will execute a demo code to demonstrate a dynamic spinlock with the irqsave feature in a kernel module on the Raspberry Pi Zero W2.

1. Prerequisites for kernel module compilation on Raspberry Pi

     $ sudo apt update
     $ sudo apt install raspberrypi-linux-headers
   

2. Sample code (filename: kernel_spinlock_irqsave_example.c)

    #include <linux/module.h>
    #include <linux/kernel.h>
    #include <linux/init.h>
    #include <linux/slab.h>
    #include <linux/spinlock.h>
   
    struct dummy_struct {
        spinlock_t lock;
        unsigned int foo;
    };
   
    static struct dummy_struct *ds;
   
    static struct dummy_struct *demo_alloc_init_function(void) {
        struct dummy_struct *ds;
        ds = kmalloc(sizeof(struct dummy_struct), GFP_KERNEL);
        if (!ds)
            return NULL;
        spin_lock_init(&ds->lock);
        return ds;
    }
   
    static int __init my_module_init(void)
    {
        unsigned long flags;
   
        pr_info("Initializing Spinlock Example Module\n");
        ds = demo_alloc_init_function();
        if (!ds)
        {
            pr_alert("Allocation failed\n");
            return -ENOMEM;
        }
   
        // Lock the spinlock and save the interrupt state
        spin_lock_irqsave(&ds->lock, flags);
        pr_info("Spinlock acquired and interrupts disabled\n");
   
        // Simulate a critical section
        ds->foo = 42;
        pr_info("Critical section executed, foo = %u\n", ds->foo);
   
        // Unlock the spinlock and restore the interrupt state
        spin_unlock_irqrestore(&ds->lock, flags);
        pr_info("Spinlock released and interrupts restored\n");
   
        return 0;
    }
   
    static void __exit my_module_exit(void)
    {
        pr_info("Exiting Spinlock Example Module\n");
        // Free the allocated memory
        if (ds)
        {
            kfree(ds);
            pr_info("Memory freed\n");
        }
    }
   
    module_init(my_module_init);
    module_exit(my_module_exit);
   
    MODULE_LICENSE("GPL");
    MODULE_AUTHOR("Suyog B <suyogburadkar@gmail.com>");
    MODULE_DESCRIPTION("An example kernel module using spin_lock_irqsave and spin_unlock_irqrestore");
    MODULE_VERSION("1.0");
   

3. Makefile

     obj-m += kernel_spinlock_irqsave_example.o
   
     all:
         make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
     clean:
         make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
   

4. Kernel module building (Out of tree module building)

    pi@SuyogB:~/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations $ make
    make -C /lib/modules/6.1.21-v8+/build M=/home/pi/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations modules
    make[1]: Entering directory '/usr/src/linux-headers-6.1.21-v8+'
      CC [M]  /home/pi/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations/kernel_spinlock_irqsave_example.o
      MODPOST /home/pi/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations/Module.symvers
      CC [M]  /home/pi/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations/kernel_spinlock_irqsave_example.mod.o
      LD [M]  /home/pi/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations/kernel_spinlock_irqsave_example.ko
    make[1]: Leaving directory '/usr/src/linux-headers-6.1.21-v8+'
   
    pi@SuyogB:~/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations $ ls -ls
    total 40
    4 -rw-r--r-- 1 pi pi 1631 Aug  3 23:48 kernel_spinlock_irqsave_example.c
    8 -rw-r--r-- 1 pi pi 7520 Aug  3 23:51 kernel_spinlock_irqsave_example.ko
    4 -rw-r--r-- 1 pi pi  121 Aug  3 23:51 kernel_spinlock_irqsave_example.mod
    4 -rw-r--r-- 1 pi pi 1007 Aug  3 23:51 kernel_spinlock_irqsave_example.mod.c
    4 -rw-r--r-- 1 pi pi 3600 Aug  3 23:51 kernel_spinlock_irqsave_example.mod.o
    8 -rw-r--r-- 1 pi pi 4656 Aug  3 23:51 kernel_spinlock_irqsave_example.o
    4 -rw-r--r-- 1 pi pi  181 Aug  3 23:47 Makefile
    4 -rw-r--r-- 1 pi pi  122 Aug  3 23:51 modules.order
    0 -rw-r--r-- 1 pi pi    0 Aug  3 23:51 Module.symvers
   

   As we can see, our kernel module has been compiled successfully. The .ko file is present, indicating that the kernel build process was executed correctly.

   Let's insert the .ko module and check the results:

5. Insert kernel object and check the results

    pi@SuyogB:~/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations $ sudo insmod kernel_spinlock_irqsave_example.ko
   
    pi@SuyogB:~/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations $ dmesg | tail
    [ 4000.476967] kernel_spinlock_irqsave_example: loading out-of-tree module taints kernel.
    [ 4000.477829] Initializing Spinlock Example Module
    [ 4000.477849] Spinlock acquired and interrupts disabled
    [ 4000.477854] Critical section executed, foo = 42
    [ 4000.477863] Spinlock released and interrupts restored
   

   The kernel module has been successfully loaded and executed. The spinlock was acquired and interrupts were disabled, allowing the critical section to run safely and update the variable foo to 42.

6. Cleaning the module

    pi@SuyogB:~/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations $ make clean
    make -C /lib/modules/6.1.21-v8+/build M=/home/pi/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations clean
    make[1]: Entering directory '/usr/src/linux-headers-6.1.21-v8+'
      CLEAN   /home/pi/linux-kernel-development-for-embedded-systems/Chapter_2_Spinlocks_Limitations/Module.symvers
    make[1]: Leaving directory '/usr/src/linux-headers-6.1.21-v8+'
   

Github Repo (Chapter 2 Limitations of spinlocks and Solution) :

Chapter 2 Github Repo Link

Now that we understand spinlocks and their details, we'll look at mutexes, our next locking tool, in the next chapter. Stay tuned!