Introduction: Is Python Compiled or Interpreted?

Python is often labeled an "interpreted" language, but this oversimplification misses its hybrid nature. In reality, Python combines compilation and interpretation to execute code. Let’s dive into how Python transforms your code into something a computer understands and why this process makes Python both flexible and efficient.

Step 1: Compilation to Bytecode

When you run a Python script (e.g., python script.py), the first step is compilation. The Python interpreter compiles your human-readable .py source code into a lower-level, platform-independent format called bytecode.

What is Bytecode?
Bytecode is a set of instructions for the Python Virtual Machine (PVM). It’s not machine code (binary instructions for your CPU), but a compact, intermediate representation of your code. Think of it as a "middle language" between Python and machine code.
Why Use Bytecode?
- Platform Independence: Bytecode isn’t tied to a specific OS or hardware.
- Efficiency: Skipping the parsing and compilation steps on subsequent runs speeds up execution.

Step 2: The Python Virtual Machine (PVM)

The compiled bytecode is executed by the Python Virtual Machine (PVM), the runtime engine of Python.

How the PVM Works
The PVM is an interpreter that reads bytecode line-by-line and executes corresponding operations. Unlike a true compiler (e.g., for C/C++), it doesn’t convert bytecode to machine code ahead of time. Instead, it dynamically interprets the bytecode during runtime.
- Myth Busting: The PVM does not convert bytecode to machine code. It interprets bytecode directly.
- Performance: While interpretation adds overhead, optimizations like bytecode caching (.pyc files) mitigate this.

Why Bytecode Executes Faster

Even though Python isn’t fully compiled, bytecode offers speed advantages:

Reduced Parsing Overhead: Your source code is parsed and compiled once (into bytecode), avoiding repetitive parsing.
Caching: Imported modules are stored as .pyc files (more on this below), skipping recompilation.

.pyc Files and the pycache Directory

Python caches bytecode in .pyc files to optimize performance. Here’s how it works:

When Are .pyc Files Created?
- When a module is imported, Python generates a .pyc file and stores it in the __pycache__ directory.
- Top-level scripts (e.g., python main.py) do not generate .pyc files by default. Only imported modules do.
Structure of pycache
The __pycache__ directory stores .pyc files with names indicating the Python version (e.g., module.cpython-310.pyc). This ensures compatibility across Python versions.
Benefits of .pyc Files
- Faster Startup: Loading cached bytecode is quicker than recompiling.
- Efficiency: Reduces redundant work in large projects with frequent imports.

Frozen Binaries: A Misconception

The term "frozen binaries" sometimes causes confusion. While .pyc files are compiled bytecode, frozen binaries typically refer to standalone executables created by tools like PyInstaller or cx_Freeze. These tools bundle the Python interpreter, bytecode, and dependencies into a single binary for distribution.

Key Difference:
.pyc files require the Python runtime, while frozen binaries include everything needed to run independently.

Why Doesn’t Python Save .pyc Files for Top-Level Scripts?

When you execute python script.py directly:

Python compiles it to bytecode but doesn’t save it to __pycache__.
The assumption is that top-level scripts may change frequently (e.g., during development), so caching offers limited benefit.

To force bytecode generation for a script, use the -m module flag or third-party tools.

Bytecode’s Platform Independence: A Double-Edged Sword

While bytecode works across platforms, the PVM is platform-specific. For example:

A .pyc file from Windows can run on Linux if the Python versions match.
However, platform-dependent operations (e.g., file paths) may still cause issues.

Conclusion: Python’s Clever Balance

Python’s compilation to bytecode and interpretation via the PVM strike a balance between flexibility and performance. By caching bytecode in .pyc files, Python speeds up execution while retaining its platform-agnostic nature. Understanding this process helps developers optimize their workflows and debug tricky issues.

Next time you see a __pycache__ folder, remember: it’s Python’s way of making your code run faster, one byte at a time!

Further Reading:

Understanding Python’s Compilation Process: Bytecode, PVM, and .pyc Files