Introduction

Applications often need to do many things at once: download files, process images, serve web requests, or compute huge datasets. This is where concurrency and parallelism come in.

While Python supports both, it has some surprising quirks due to a core component called the Global Interpreter Lock (GIL).

This blog dives into:

• How multithreading and multiprocessing work in Python

• Why Python has the GIL

• How it compares with languages like Go, Rust, and Java

• When to use which model, with code examples

Key Terms You Should Know First

Before diving deep into Python’s multithreading and multiprocessing behaviour, let’s clarify some commonly confused terms. These concepts often overlap but have subtle and important differences.

Concurrency

“Doing many things at once, but not necessarily simultaneously.”

• Concurrency is about managing multiple tasks so they all make progress.

• Even on a single CPU, tasks can take turns via time-slicing.

• Great for I/O-bound operations (like waiting for disk or network).

Parallelism

“Actually doing many things at the same time.”

• Requires multiple CPU cores.

• Great for CPU-bound tasks like heavy computations.

• Achieved by running tasks in separate CPU cores simultaneously.

Process

A program in execution, with its own memory, resources, and runtime context.

• A process is the largest unit of execution in an operating system.

• It runs independently and does not share memory with other processes.

• One process can contain one or more threads.

Thread

A lightweight unit of execution inside a process.

• Shares memory with other threads in the same process.

• Faster to create and switch between than full processes.

• In Python, only one thread runs at a time (due to the GIL), so it's mostly used for I/O-bound concurrency.

Multithreading

Running multiple threads within a single process.

• Ideal for I/O-bound workloads (e.g., web requests, file access).

• In Python, limited by the GIL, so no true parallel execution for CPU-bound code.

Multiprocessing

Running multiple independent processes.

• Each process has its own memory space and its own GIL, allowing true parallelism.

• Best for CPU-bound tasks like computation-heavy workloads.

• Comes with overhead (more memory, slower communication).

In Python:

• Multithreading enables concurrency, but not parallelism (due to the GIL).

• Multiprocessing enables parallelism by using separate processes.

What is the GIL, and Why Does Python Have It?

The Global Interpreter Lock (GIL) is a mutex, a special lock used to prevent multiple threads from accessing shared resources simultaneously. In Python, it ensures only one thread runs Python bytecode at a time.

Why does it exist?

1. Reference counting is not thread-safe.

   • Python uses reference counts for memory management. (A reference is a pointer or link from one object to another)

   • Without the GIL, every increment/decrement would need to be atomic or use a lock.

2. Simplicity.

   • GIL made interpreter design easier and faster when Python was created.

3. C extension compatibility.

   • Libraries like NumPy assume single-threaded access to Python objects.

Trade-off:

• Simpler and safer interpreter

• CPU-bound tasks cannot run in parallel with threads

Multi-Threading in Python

Python has a threading module that allows multiple threads to run. But due to the GIL, only one thread executes Python bytecode at a time, even on a multi-core CPU.

Use when:

• Tasks are I/O-bound (networking, disk access)

• You want lightweight concurrent tasks (e.g., background logging)

Example:

import threading
import time

def print_numbers():
    for i in range(5):
        print(f"Number: {i}")
        time.sleep(1)

def print_letters():
    for letter in 'abcde':
        print(f"Letter: {letter}")
        time.sleep(1)

t1 = threading.Thread(target=print_numbers)
t2 = threading.Thread(target=print_letters)

t1.start()
t2.start()

t1.join()
t2.join()

Output:

⛔ Why no true parallelism?

Because of the Global Interpreter Lock (GIL), only one thread can execute Python code at a time, even on multi-core machines.

Multi-Processing in Python

Python’s multiprocessing module works by spawning separate processes, each with its own Python interpreter and GIL. So they can truly run in parallel on multiple cores.

Use when:

• Tasks are CPU-bound

• You want to utilise all CPU cores

Example:

from multiprocessing import Process
import time

def compute():
    print("Computing...")
    time.sleep(2)
    print("Done")

if __name__ == '__main__':
    p1 = Process(target=compute)
    p2 = Process(target=compute)

    p1.start()
    p2.start()

    p1.join()
    p2.join()

Output:

⚠️ Downsides:

• Heavier than threads (more memory and startup time)

• Requires pickling/unpickling to send data between processes

• Data sharing is clunky (queues, pipes, shared memory)

What is Pickling?

Pickling refers to serialising Python objects into byte streams. It is used in multiprocessing to send data between processes:

• One process "pickles" the object. (serialises the data)

• It’s sent via a pipe or queue.

• The receiving process "unpickles" it. (deserialises the data)

import pickle

data = {'name': 'Pixel22', 'age': 22}

print("Before Pickling: ", data)

# Serialize to bytes
pickled_data = pickle.dumps(data)

print("Pickled Data: ", pickled_data)

# Deserialize back to object
original = pickle.loads(pickled_data)

print("Deserialized: ", original)

Output:

Before Pickiling:  {'name': 'Pixel22', 'age': 22}
Pickled Data:  b'\x80\x04\x95\x1e\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x04name\x94\x8c\x07Pixel22\x94\x8c\x03age\x94K\x16u.'
Deserialized:  {'name': 'Pixel22', 'age': 22}

This isn’t needed in multithreading because threads share memory. But multiprocessing isolates memory, so we must use serialisation.

How Other Languages Compare

Python’s concurrency is constrained by the GIL, so multiprocessing is required for CPU-bound parallelism. Other languages don’t need that workaround.

Limitations in Real-World Python Concurrency

Here’s where Python concurrency can break down:

• Trying to use threads for CPU-bound tasks → no speedup

• Sharing large data between processes → serialisation overhead

• Building scalable async systems → harder than Go or Rust

What’s the Future?

PEP 703: Making the GIL Optional

There’s an effort to make a no-GIL version of Python (currently experimental in Python 3.13+). It could allow real multi-threaded Python code — but it’ll take time and ecosystem-wide changes.

Final Thoughts

Some key takeaways to help you choose the right approach depending on your workload:

• Use multithreading for I/O-bound tasks.

• Use multiprocessing for CPU-bound tasks.

• Know that Python’s GIL is a historical trade-off.

• Other languages don’t need multiprocessing for parallelism; their threading is stronger because they were designed that way from the start.