Troubleshooting is a Lifestyle 😎

I was reading Rodrigo's blog the other day, and came across a great point in his How I prepare a technical talk article. He says:

I don't script my talks, but I typically write a blog article that I use as a reference for my talk.

I love the idea - but I already gave this talk at DjangoCon US 2024 - so now I'm doing the reverse.

---

We all deal with challenging problems that sometimes seem impossible to solve, whether it's a Django error that makes no sense, an electronic device that won't power on, or a network connection that keeps dropping packets. In those moments, the difference between frustration and success often comes down to the way we troubleshoot the issue.

I've spent years programming, maintaining radar systems, and designing electronics, and if there's one truth I've discovered, it's that troubleshooting isn't just a skill - it's a mindset. A lifestyle even!

I'm the main programmer and managing director of Watervize, where we build data systems for irrigation districts. When I'm not working on the business, I serve as a Radar Maintenance Officer in the Marine Corps, supervising a team that keeps complex radar systems running. Between these roles, my background in electronics design, and my experience with Django development over the past six years, I've had plenty of opportunities to troubleshoot in various fields - from mechanical and power systems to digital electronics, RF systems, embedded software, and web applications.

I won't make you an expert troubleshooter in the time it takes to read this article. You may even already be an excellent troubleshooter. My goal is to give beginners some new tools for their toolkit, and to help experienced folks think through aspects of troubleshooting they may have forgotten, or may never have actually considered. At the end of the day, troubleshooting is a skill, and like any skill, the more you practice it and the better the tools you have, the more effective you'll become at it.

The Troubleshooting Mindset

Before we dive into practical techniques, let's clarify some terminology to add context for the discussion:

Problem solving is a broad term that covers any approach to resolving a problem or challenge. This could be a technical issue, a conceptual puzzle, or even a personal dilemma.

Troubleshooting is a subset of problem solving. It follows a specific pattern: recognizing that there's a problem, locating the cause of that problem, and then resolving it. It typically applies to tangible systems like hardware, software, or mechanical equipment.

Debugging is essentially troubleshooting specific to software and programming. When you're hunting down an elusive bug in your Django app, you're debugging.

A Cautionary Tale

Let me tell you about a bad troubleshooting experience I witnessed years ago. I was responsible for evaluating a team who were setting up a communication site in the desert. Their one goal was to establish communication to another location on an airfield not far away.

When I arrived, I discovered they had erected not one, not two, but four different antennas. They only needed to establish a single link, but had assembled four very different types of antennas - a loop antenna, a sloping-V, and others.

When I asked what was happening, they explained, "The first one we set up didn't work, so we moved on to set up a second one..." and so on. Looking at the antennas, I noticed obvious issues - incorrect wire lengths, missing grounding, and other fundamental problems that could have been identified with basic troubleshooting.

I asked the person in charge, "What are you using to build these antennas? Where are your manuals?" He looked at me with a completely deadpan face and replied, "Sir, I don't need manuals. I rely on experience!"

🤦

This is a classic example of what's called "shotgun troubleshooting" or "shotgun debugging." If you're familiar with marksmanship, you know a shotgun tends to scatter pellets widely. Similarly, this approach involves making changes randomly without a systematic approach, hoping something will 'hit the target' and solve the issue. It's inefficient and often ineffective.

With the right tools, techniques, and indicators, there's always a systematic way to troubleshoot a problem - whether it's in the Django web framework or any other field.

Troubleshooting is a Universal Skill

We all start learning troubleshooting at an early age. Consider the children's book "Curious George on a bike" - when George bends his bicycle wheel after crashing into a rock, he identifies the problem (the bent wheel), and devises a solution - riding a wheelie! This is the sort of basic troubleshooting we all learn as children.

Troubleshooting is everywhere in our lives:

• Medicine: Doctors use symptoms, tests, and medical knowledge to diagnose and treat health problems

• Aviation: Pilots and maintenance crews must quickly identify and resolve problems to ensure safety

• Military: Technical specialists maintain complex equipment in challenging environments

• Tech: Developers and engineers trace issues through multiple layers of software, hardware, and networks

The most successful troubleshooters share two essential qualities: persistence and curiosity. Some problems are easy to solve, others not so much. If you give up easily or aren't curious enough to explore different paths, you'll struggle to resolve complex issues.

Indicators and Resources

When it comes to the practical side of troubleshooting, the first key element is understanding what indicators - anything that can provide information about the context of the problem you're trying to solve - are available.

As Abraham Maslow once said:

"If your only tool is a hammer, then every problem looks like a nail." If you're using the same indicators for all your problems, or just sticking with the basics, you'll have limited context when tackling deeper issues.

Types of Indicators

We can categorize indicators in several ways:

1. Built-ins vs. Add-ons: Systems come with built-in indicators, and we can often add more to gain additional insights.

2. Alerts vs. Status: Some indicators proactively tell you when something's wrong, while others require you to check their current state.

3. Qualitative vs. Quantitative: Some indicators give you a general sense that something is good or bad, while others provide specific measurable values.

4. Affirmative vs. Negative: Some indicators tell you when something is working fine, others when something is wrong.

Indicators in Everyday Life

This concept applies to everything around us. Take a car, for example:

• Built-in: Dashboard lights, gauges, warning sounds

• Add-ons: OBD scanners, tire pressure gauges, diagnostic tools

Maybe an odd example, but to show that it applies to just about anything: houseplants have indicators:

• Built-in: Leaf color, wilting, soil moisture

• Add-ons: Moisture meters, temperature sensors

Even my dog, Lady Duchess, has her indicators:

• Built-in: Her level of cuddliness (if she's unusually cuddly, something's wrong - she usually wants her space), her rate of treat consumption (if she's not eagerly taking treats, something's definitely off)

• Add-ons: Her GPS collar that tracks steps, rest, and other activities

The point is, anything we might need to troubleshoot has built-in indicators we can observe, and often we can add indicators that provide additional context.

Django Indicators and Tools

The same principles about indicators apply perfectly to troubleshooting Django applications. Let's look at what's available to us as Django developers.

Built-in Django Indicators

Django comes with several powerful troubleshooting tools right out of the box:

Template Error Pages: When DEBUG=True in your settings, Django displays detailed error pages that show the exact error message, what Django tried to do, and suggestions for fixing the issue. If you're using the Django-Extensions package, you can also use Werkzeug, which gives you an interactive debugging console right in the browser.

# settings.py
DEBUG = True  # Enables detailed error pages during development

Error Reporting Via Email: When DEBUG=False on your production or staging systems, Django can email you the same information that would appear on the error page:

# settings.py
DEBUG = False
ADMINS = [
    ("Mary", "mary@example.com"),
    ("John", "john@example.com"),
]

The Checks Framework: This powerful but often overlooked tool runs a set of static checks when you start your server to verify that your configuration is correct. It examines your database setup, models, security settings, templates, URLs, and more:

python manage.py check

The checks framework will tell you if something is an error, a warning, or just informational.

It's also extensible - you can add custom checks for your specific application requirements. Here's a quick example of adding a custom check:

# yourapp/checks.py
from django.core.checks import register, Warning

@register()
def example_check(app_configs, **kwargs):
    errors = []
    # Check something important for your app
    if some_condition_is_not_met:
        errors.append(
            Warning(
                'An important setting is misconfigured',
                hint='Try setting IMPORTANT_SETTING=True in settings.py',
                id='yourapp.W001',
            )
        )
    return errors

(Also, take a look at the django-extra-checks package for more helpful additions to the checks framework)

Console and Logging: Django's shell and logging system provide invaluable context about what's happening in your application. You can increase the verbosity of most Django management commands with the -v flag:

python manage.py migrate -v 3  # Show detailed migration information

Add-on Tools for Django

When the built-in tools aren't enough, there's an entire ecosystem of add-on packages that can enhance your troubleshooting:

Django Debug Toolbar: This oft-recommended tool adds panels to your pages showing detailed information about execution time, database queries, template rendering, cache usage, and more. It's configurable and extensible too.

Performance Profilers: Tools like Django Silk can help identify performance bottlenecks by showing you exactly which views and database queries are slowing down your application.

Error Tracking and Performance Monitoring: Services like Sentry, Rollbar, New Relic, and others provide detailed error reporting and performance metrics. They capture the full traceback, request information, user context, performance details, and more. They can alert you when errors occur and help you prioritize issues based on frequency and impact.

For example, with Sentry integrated, you'll receive detailed error reports including:

• The exact error message

• Context around the event

• The complete stack trace

• Database and cache information

• User information (if available)

• Tags you can add to organize errors

Health Checks: Packages like django-health-check or django-watchman can monitor the health of your application components:

# urls.py
urlpatterns = [
    path('health/', include('health_check.urls')),
]

This gives you a quick dashboard showing the status of your database connections, Celery tasks, Redis, and other services your application depends on.

The package also simplifies the process of adding custom health checks.

Custom Admin Panels: You can also add custom panels to your project’s admin to monitor specific aspects of your application. For example, I once had intermittent issues with Celery processing tasks, so I created a simple admin panel that showed task processing rates compared to historical averages:

@admin.register(CeleryTaskStats)
class CeleryTaskStatsAdmin(admin.ModelAdmin):
    change_list_template = 'admin/celery_task_stats_change_list.html'

    def changelist_view(self, request, extra_context=None):
        extra_context = extra_context or {}

        # Get stats for current hour, day, week
        current_hour = get_task_count(hours=1)
        current_day = get_task_count(hours=24)
        current_week = get_task_count(days=7)

        # Get stats for previous periods
        previous_hour = get_task_count(hours=1, offset_hours=1)
        previous_day = get_task_count(hours=24, offset_hours=24)
        previous_week = get_task_count(days=7, offset_days=7)

        extra_context['task_stats'] = {
            'current': {
                'hour': current_hour,
                'day': current_day,
                'week': current_week,
            },
            'previous': {
                'hour': previous_hour,
                'day': previous_day,
                'week': previous_week,
            }
        }

        return super().changelist_view(request, extra_context=extra_context)

Breaking Down Problems

Once you've identified that there is a problem through indicators, the next step is breaking it down into manageable pieces.

Any big, challenging problem can be broken into a set of smaller, simpler problems you can solve.

Isolating Variables and Narrowing Scope

When faced with a complex system like a Django application, trying to hold the entire system in your head while troubleshooting is nearly impossible. You need to find ways to narrow the scope and isolate variables to identify exactly where the problem lies.

Here are some strategies for breaking down problems:

Make Your System Testable: Writing testable code naturally leads to better troubleshooting. When you make functions and methods idempotent (same input always produces same output), keep code blocks reasonably small, and focus each function on a single responsibility, you make it much easier to isolate and test components separately.

Use Breakpoints Effectively: Debugging has a long, interesting history. Betty Holberton, who worked on the ENIAC (one of the first digital computers), developed the concept of "breakpoints" by literally pulling cables to stop program execution at specific points. This fundamental technique is something all troubleshooters still use today - stopping execution to examine the state of your application at critical points.

In Django, you can use Python's debugger (pdb) or your IDE's debugging tools to set breakpoints and inspect variables:

def my_view(request):
    # Set a breakpoint
    import pdb
    pdb.set_trace()

    # Or using the new built-in in Python 3.7+
    breakpoint()

    # Your code continues...
    result = some_calculation(request.GET.get('value'))
    return render(request, 'template.html', {'result': result})

When execution reaches the breakpoint, you can examine variables, execute code line by line, and understand exactly what's happening at that moment.

Extract and Test Components: When you suspect a particular function or method is causing problems, extract it and test it separately:

# In your shell or a test file
from myapp.utils import problematic_function

# Test with different inputs
result1 = problematic_function(input1)
result2 = problematic_function(input2)

# Compare expected vs actual results

This approach helps you confirm whether the problem is in this specific function or elsewhere in your application.

Testing: While outside the scope of this article, it's always good to remember that proper testing (unit testing with pytest or unittest, along with integration testing and end-to-end testing) can reduce the amount of troubleshooting you end up doing later on as your codebase evolves.

Balance Narrowing Scope with Avoiding Tunnel Vision

While narrowing the scope is crucial, you don't want to prematurely focus on a specific function or line of code that might not be the actual source of the problem. It's easy to get fixated on something and spend hours troubleshooting a small block of code that turns out not to be the issue.

To avoid tunnel vision, periodically ask yourself:

• Did I make the correct assumptions when I started narrowing my scope?

• Did I miss something important that might have led me elsewhere?

• Am I considering all of the available indicators?

• Is the problem really what I think it is, and is it where I think it is?

• Could there be more than one problem happening simultaneously? 😱

Having multiple problems at once is particularly challenging, which is why having well-structured, testable code is so valuable. It allows you to verify each component independently.

Getting an outside perspective can be invaluable when you've been staring at the same code for hours. Sometimes, simply explaining the problem to someone else can lead to insights. This technique is often called "rubber duck debugging" - the act of explaining your problem to an inanimate object (traditionally a rubber duck) forces you to articulate the issue clearly, which often reveals the solution.

If you don't have a rubber duck handy, you can use a regionally appropriate animal - for example, Random Geek from New Mexico recommends a Javelina!

When and How to Ask for Help

There comes a point in every troubleshooting journey where you might need to reach out for assistance. Knowing when and how to ask for help can make all the difference.

Knowing When to Seek Assistance

Consider asking for help when:

• You're hopelessly stuck and frustrated: When stress levels increase, thinking quality decreases. Sometimes stepping away or seeking help is the most productive thing you can do.

• You want new perspectives: Sometimes you're on the right track but could benefit from someone who's seen similar issues before.

• You need a fresh pair of eyes: We all get tired and miss things that might be obvious to someone coming in fresh.

Remember, asking questions isn't a sign of failure - it's a skill in itself, and one worth developing. The ability to clearly communicate technical problems is incredibly valuable in development work.

How to Ask Effective Questions

For your questions to be answered effectively, you need to:

• Provide context: Explain the problem clearly, including how it started and what you've observed.

• Be specific: Share relevant code snippets and error messages. Don't assume the problem is obvious to others - even experienced developers need context when looking at unfamiliar code.

• Show what you've tried: Demonstrating the steps you've already taken not only prevents redundant suggestions but shows respect for the time of those helping you.

• Be open to suggestions: Sometimes the best solutions come from unexpected directions or approaches you hadn't considered.

Creating a Minimal Reproducible Example (MRE)

One of the most effective ways to get help is to provide a minimal reproducible example (MRE) - the smallest amount of code and context needed for someone else to reproduce your issue.

A bad example:

"I'm using Django and I want to upload a file. I've created a model and a form, but I get errors when I try to upload a file."

What's missing here? The version of Django, the specific error message, code snippets, and what the person has already tried.

A better example:

"I'm using Django 5.0 and I'm trying to create a form to upload a file. I created a model with a FileField and a form with a FileField, but when I go to the view in my browser, the form isn't shown. Here's my code:

models.py:
class MyModel(models.Model):
    file = models.FileField()

forms.py:
class MyForm(forms.ModelForm):
    class Meta:
        model = MyModel
        fields = ['file']

views.py:
def my_view(request):
    template = 'my_template.html'
    if request.method == 'POST':
        form = MyForm(request.POST, request.FILES)
        if form.is_valid():
            form.save()
            return redirect('my_view')
    else:
        form = MyForm()
    return render(request, template, {'form': form})

<form method="post" enctype="multipart/form-data">
    {% csrf_token %}
    {{ forms }}
    <button type="submit">Submit</button>
</form>

Now someone can quickly spot the issue - a typo in the template using {{ forms }} instead of {{ form }}. This is the kind of clear, specific example that gets quick, accurate help.

Where to Get Django Help

When you need help with Django specifically, there are several excellent resources:

• The "Using Django" section of the Django Forum

• The Django Ticket Tracker

• Social media platforms like Mastodon (and other places)

• Reddit communities like r/django and r/djangolearning

• Stack Overflow (with various Django-related tags)

• Django Discord

• #Django IRC channel on Libera.chat

Documenting the Process

The final critical component of effective troubleshooting is documentation. Writing down your process helps not only you in the future but also others who might encounter similar issues.

Why Documentation Matters

Documentation is especially important (and a kindness to future-you) when:

• You're working on a long-running or complex problem that might span multiple sessions

• The solution might be useful to others on your team or in the broader community

• The issue could recur, and you want to save future-you the trouble of rediscovering the solution

What to Document

An effective troubleshooting log includes:

• What was the problem? Describe the issue clearly, including any error messages or unusual behavior.

• What did you try? Document each approach you took, whether successful or not.

• Why did you try that? The reasoning behind your choices is often as valuable as the steps themselves.

• What did you expect to see? Your assumptions and expectations are important context.

• What was the outcome? The actual results of each attempted solution.

• What else might you try? If you had to stop troubleshooting, note your thoughts about potential next steps.

Writing down what else you were thinking about trying is particularly valuable when you need to step away from a problem and return to it later. Human memory is in short supply, and without documentation, you might lose valuable insights or approaches you had considered.

Sharing Your Solutions

Whenever possible, share your solutions with the wider community. This might be:

• A detailed comment in your commit message

• A post on Stack Overflow

• A "Today I Learned" blog post

• Documentation in your project wiki

• A contribution to official documentation

Remember, if you struggled with something, others probably will too. Your documentation could save someone (or future-you!) hours or days of frustration.

Review and Conclusion

Let's go over the key points to consider when making troubleshooting a lifestyle:

1. Understand and use indicators and tools: Learn what's available to help you identify problems and narrow down their causes. In Django, this includes error pages, the checks framework, logging, debug toolbar, and monitoring services like Sentry.

2. Break problems down into manageable chunks: Complex systems are made up of simpler components. Test them separately to isolate issues.

3. Avoid tunnel vision: Don't get so focused on one area that you miss the bigger picture. Periodically step back and reassess your assumptions.

4. Ask for help effectively: Provide context, be specific, show what you've tried, and use minimal reproducible examples.

5. Document everything: Write down your process, findings, and solutions to benefit both yourself and others.

The more effectively you troubleshoot, the better you'll become at it. Like any skill, it improves with quality practice.

At the end of the day, troubleshooting isn't just about fixing technical problems. It's a mindset that applies across all aspects of life and work. Whether you're dealing with a Django error, a hardware malfunction, or even a communication breakdown in a team, the systematic approach of identifying the problem, narrowing down the cause, and methodically testing solutions will serve you well.

Remember that with the right tools, techniques, and indicators, there's always a way to troubleshoot a problem that makes sense and will get you to a solution. Stay persistent, maintain your curiosity, and keep refining your approach - and you'll find that troubleshooting truly becomes a lifestyle that helps you in any area of life.

---

Bonus: Actively practice your Python troubleshooting skills with Eric Matthes' py-bugger, which inserts bugs for you to resolve. An excellent way to practice!