Python Dependency Isolation with Subprocesses

Introduction

Resolving conflicting dependencies can be a real headache in python applications. Different parts of the codebase might require different versions of the same package, creating a seemingly impossible situation. While virtual environments help with project-level isolation, they don't solve the problem of conflicting dependencies within a single application. In this article, we'll explore a proof-of-concept approach that uses Python's subprocess mechanism to run specific functions with isolated dependencies. This is purely for exploration and is not at all recommended to be productionized, unless you are ready for late night pagers!

Glossary

Subprocess: A separate process spawned by a parent process, with its own memory space and environment
Runtime Environment: The set of dependencies available to a function when it executes
Dependency Isolation: Running code with specific dependencies regardless of what's installed in the parent process

The Challenge

Consider this scenario: one part of your application needs requests==2.22.0 while another requires requests==2.32.2. How do you satisfy both requirements within the same Python process? The initial thoughts are:

1. Compromise: Choose one version and hope everything works

2. Refactoring: Rewrite code to work with a single version

Let us explore a third option: run specific functions in isolated subprocesses with their own dependencies.

The Subprocess Solution

The core idea is: when a function needs specific dependencies, we:

1. Spawn a new Python subprocess

2. Install the required dependencies in a temporary directory

3. Execute the function code in this subprocess

4. Capture the result and return it to the parent process

This approach leverages the fact that subprocesses have their own memory space and Python environment, allowing for true isolation without affecting the parent process.

Key Components

Our solution consists of two main parts:

1. A decorator (runtime_env) that wraps functions to execute in isolated subprocesses

2. A subprocess runner that handles dependency installation and function execution

Here's how the decorator is used:

from decorator import runtime_env

@runtime_env("requirements.txt")
def process_data(data, factor=1):
    # This function runs in a subprocess with dependencies from requirements.txt
    import requests  # Will use version specified in requirements.txt

    # Process data...
    result = {
        "original": data,
        "processed": {"name": data["name"].upper(), "value": data["value"] * factor},
        "metadata": {"requests_version": requests.__version__}
    }

    return result

How It Works

When the decorated function is called, the following sequence happens:

1. Function identification: The decorator captures the source code of the module containing the function

2. Argument serialization: Function arguments are serialized using Python's pickle module

3. Subprocess creation: A new Python process is started, running our subprocess runner

4. Dependency installation: The subprocess installs required packages in a temporary directory

5. Function execution: The function runs in the isolated environment with its own dependencies

6. Result communication: The result is serialized and passed back to the parent process

7. Cleanup: Temporary directories and files are removed

Example in Action

Let's explore a concrete example. The parent process might have requests==2.26.0 installed, while we want a function to run with requests==2.28.0:

# Parent process
def get_data():
    try:
        import requests
        print(f"Parent process using requests version: {requests.__version__}")
    except ImportError:
        print("Requests not available in parent")
    return {"name": "Test Data", "value": 42}

# This will run in a subprocess with its own dependencies
@runtime_env("requirements.txt")  # Contains requests==2.28.0
def process_data(data):
    import requests
    print(f"Subprocess using requests version: {requests.__version__}")
    return {"result": data, "version_used": requests.__version__}

# Main execution
input_data = get_data()
result = process_data(input_data)
print(f"Process completed with requests version: {result['version_used']}")

When executed, we see different versions of the requests library being used in each process, demonstrating successful isolation. Here’s a screenshot from the PoC that we did:

As you can see in the example above, the parent process PID is 1, where as the subprocess PID is 7, and for both the parent process and subprocess, the requests library version is different.

Behind the Scenes

I won’t be delving in the exact approach here, I would let you take a stab at it, and yes, use LLMs to help you solution! However, here’s a summary of the approach at a very high level:

• The decorator first identifies the module to which the function belongs, and get the module via the sys package, and uses the inspect module to fetch the source file. We need the module in order to ensure that we can also use the imports used for the decorated function.

• Once we have the code information, we use the subprocess module to create a new process, and we create a new temporary directory in which we install the required dependencies

• Now we execute the function and return the output back to the caller process.

Only for fun, not for production!

This approach is only for educational purposes, the primary reason being, this approach is very naive, and there are a lot of aspects to cover even to make it dev ready, for example, the protocol or mechanism used to communicate the results back to the parent process. What about concurrent processes creating new isolated envs? What about common dependencies between the parent and the child process? And what not!

Explore something production ready?

To checkout an implementation that deals with conflicting dependencies you can read about Ray's Runtime Environments feature, which provides a more robust, optimized approach to dependency isolation with additional features like package caching and environment reuse.

Conclusion

Python’s subprocess is a powerful module. The example above reveals the power of process isolation as a solution to conflicting dependencies, a common challenge in modern Python development. While this is a proof-of-concept, nevertheless, understanding the underlying principles of process isolation provides valuable insight into how dependency conflicts can be managed in complex applications.

If you want to explore the source code, feel free to get in touch! However, I strongly recommend doing it yourself, with LLMs at your disposal, should be an hour of effort!