Django Assets Management: Best Practices

Introduction

I have worked with legacy Django projects in the past, and one thing I noticed is that static (css, jss, images) and media (images, videos, pdf, etc) files are often handled incorrectly. Media files were being served directly from the server, which you should never do. Instead, use a third-party service like Cloudinary, AWS S3, Cloudflare, or Google Cloud Storage to handle user-uploaded files.

In some projects, even when media files were correctly served using third-party services, engineers sometimes used these services for the application's own static files as well. While this is acceptable, it adds unnecessary complexity to your code and deployments.

You can use Nginx or Apache web server to serve your static files or user-uploaded files. While this is possible, it's generally not a good idea to serve user-uploaded files from the main application server. Static and media files consume bandwidth and can slow down your app server. This also poses security threats and can lead to malicious uploads executing on your server. Additionally, it's harder to cache efficiently compared to dedicated third-party services.

My approach is to serve my app's static files internally, making the application self-contained and less complicated. You can also serve these files behind a CDN for better performance. For user-uploaded media, the best approach is to use a third-party service. In this guide, we will use WhiteNoise for our static files backend and django-storages with AWS S3 for handling user-uploaded files. We will also take an extra step by optimizing user-uploaded images; resizing and converting to WEBP format before uploading them to the S3 bucket. This will result in much smaller image file sizes while maintaining relatively good quality for your application.

Serving Django static files with WhiteNoise

First, we need to install whitenoise in your django project:

pip install whitenoise

Next, edit your settings.py file and add WhiteNoise to the MIDDLEWARE list, above all other middleware apart from Django’s SecurityMiddleware:

MIDDLEWARE = [
    # ...
    "django.middleware.security.SecurityMiddleware",
    "whitenoise.middleware.WhiteNoiseMiddleware",
    # ...
]

Make sure you have the STATIC_URL and STATIC_ROOT settings too:

# ...
STATIC_URL = 'static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
# ...

Make sure you add staticfiles to your .gitignore file. WhiteNoise will now serve your static files 🎉.

To achieve the best performance, you should enable compression and caching in settings.py:

STORAGES = {
    # ...
    "staticfiles": {
        "BACKEND": "whitenoise.storage.CompressedManifestStaticFilesStorage",
    },
}

To check if WhiteNoise is working properly, visit any page with static files. You should also turn off debug mode by setting DEBUG=False in your Django settings.py, as Django automatically serves static files when in debug mode.

From the screenshot, you can see that the first time I visited the /admin/login page, a status code of 200 was sent, indicating that the static files had to be downloaded initially. When you refresh, it sends a status code of 304, telling the browser to use the cache. This is how WhiteNoise helps us by setting the correct headers and cache for our static files.

While WhiteNoise is efficient, combining it with a CDN like Cloudflare or Cloudfront usually provides better performance, especially for high-traffic websites. Besides speed and security improvements, you can also save money 💲.

Serving user-uploaded files with S3 Bucket

In this demo, we will work with three types of images in different models: an avatar image in the User model, a category image in the Category model, and a product image in the Product model. We will begin by storing the uploaded files in an S3 bucket. Then, we will create a custom BaseWebPS3Storage to resize the images and convert them to the webp format.

Setup S3 Bucket

Let's set up our S3 bucket on AWS. First, create a new user to manage the S3 bucket. You can use an existing user if you prefer, but I usually like to create a new user specifically for S3:

Go to IAM > Users, then click on Create user. You can name it anything; I will name mine django-s3-user. On the permissions screen, search for AmazonS3FullAccess and check it. You can also set restricted access and AWS IAM rules that fit your application.

After creating the user, click on the user to create an access key. Copy the access ID and secret, and add them as environment variables. You should also copy the user's ARN, as we will need it for the S3 bucket permissions. It usually looks like arn:aws:iam::your-AWS-account-ID:user/your-user-name. For security and best practices, make sure not to use the root account's credentials to create access keys.

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_STORAGE_BUCKET_NAME=

Head to Amazon S3, and create a new bucket:

You can block public access if you only want access within your app.

Click on the newly created S3 bucket and edit it’s Permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor01",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::your-AWS-account-ID:user/django-s3-user"
            },
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::django-staticfiles/*",
                "arn:aws:s3:::django-staticfiles"
            ]
        }
    ]
}

Install and configure the necessary packages in Django

pip install "django-storages[s3]"

This will install the major packages like django-storages, boto3, and botocore.

Next, edit STORAGES in settings.py :

AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')

STORAGES = {
  "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
    },
    # ...
}

Now, when users upload their files, they will be uploaded to our S3 bucket.

Optimizing Image Upload with Pillow

One thing that greatly affects an application's loading time is media files. For instance, letting users upload an image that is 4000 × 3000 for an avatar doesn't make sense when we only show it at 220 × 200 dimensions, or saving a category image when we only need it at 500 × 500. In this demonstration, we will create custom storage backends to save avatar, product, and category images by resizing and converting them to the WEBP format. I have written about WEBP images in this blog post, which you can check out.

Most Django apps will already have Pillow installed, but if you don’t have it, you can install it with:

pip install pillow

Next, let’s create our storage_backends.py by creating the BaseWebPS3Storage class that other backend would inherit from:

from storages.backends.s3 import S3Storage
from PIL import Image
import io
import os
from typing import Tuple, Optional

class BaseWebPS3Storage(S3Storage):
    """Base storage class with common WebP conversion logic"""
    MAX_DIMENSIONS: Tuple[int, int] = (1920, 1080)  # (width, height)
    QUALITY: int = 80
    ALLOWED_EXTENSIONS = ('.png', '.jpg', '.jpeg', '.webp')

    def _convert_to_webp(self, image: Image.Image, quality: Optional[int] = None) -> io.BytesIO:
        """Convert image to WebP format with specified quality"""
        if image.mode in ('RGBA', 'LA') or (image.mode == 'P' and 'transparency' in image.info):
            image = image.convert('RGB')

        webp_content = io.BytesIO()
        image.save(
            webp_content, 
            format='WEBP', 
            quality=quality or self.QUALITY,
            method=6
        )
        webp_content.seek(0)
        return webp_content

    def _resize_image(self, image: Image.Image) -> Image.Image:
        """Resize image if it exceeds maximum dimensions"""
        max_width, max_height = self.MAX_DIMENSIONS
        if image.width > max_width or image.height > max_height:
            image.thumbnail((max_width, max_height), Image.Resampling.LANCZOS)
        return image

    def _process_image(self, name: str, content: io.BytesIO) -> Tuple[str, io.BytesIO]:
        """Process image by resizing and converting to WebP"""
        if not name.lower().endswith(self.ALLOWED_EXTENSIONS):
            return name, content
        try:
            image = Image.open(content)
            image = self._resize_image(image)
            content = self._convert_to_webp(image)
            name = f"{os.path.splitext(name)[0]}.webp"
        except Exception as e:
            print(f"Failed to process {self.__class__.__name__} image {name}: {str(e)}")
        return name, content

    def _save(self, name: str, content: io.BytesIO) -> str:
        """Save the file, processing it if it's an image"""
        if not name:
            raise ValueError("File name is required")

        name, content = self._process_image(name, content)
        return super()._save(name, content)

Our BaseWebPS3Storage sets a default maximum dimension of 1920 x 1080 and a default image quality of 80%. The class attempts to resize the image if its dimensions exceed the default. You should adjust these values to fit your application's needs.

Next, we will create the DefaultWebPS3Storage class:

class DefaultWebPS3Storage(BaseWebPS3Storage):
    """Default storage backend for all images"""
    QUALITY = 75

We can now replace the S3Storage backend class in our STORAGES with our custom class:

STORAGES = {
  "default": {
        "BACKEND": "django_assets.storage_backends.DefaultWebPS3Storage",
    },
    # ...
}

django_assets.storage_backends.DefaultWebPS3Storage is the path to where the class is located in your project. I usually place it in the same folder as my settings.py file or in a utils folder.

Adding multiple Storage backends for different Images

Let’s say you have Product, Category and Avatar in different Django models, it make sense to use different storage backend with custom inputs:

class UserAvatarStorage(BaseWebPS3Storage):
    """Storage backend for user avatars"""
    MAX_DIMENSIONS = (240, 200)
    QUALITY = 85

class CategoryImageStorage(BaseWebPS3Storage):
    """Storage backend for category images"""
    MAX_DIMENSIONS = (800, 800)
    QUALITY = 90

class ProductImageStorage(BaseWebPS3Storage):
    """Storage backend for product images"""
    MAX_DIMENSIONS = (1024, 840)
    QUALITY = 75

Django's models.ImageField lets us use custom backend storage. We can apply them in our models like this:

from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils.translation import gettext_lazy as _
from django_assets.storage_backends import UserAvatarStorage, ProductImageStorage, CategoryImageStorage

class User(AbstractUser):
    # ...
    avatar = models.ImageField(
        _('avatar'),
        upload_to='avatars/',
        null=True,
        blank=True,
        storage=UserAvatarStorage
    )

class Product(models.Model):
    # ...
    image = models.ImageField(
        _('image'),
        upload_to='products/',
        null=True,
        blank=True,
        storage=ProductImageStorage
    )

class Category(models.Model):
    # ...
    image = models.ImageField(
        _('image'),
        upload_to='categories/',
        null=True,
        blank=True,
        storage=CategoryImageStorage
    )

That's it! We can easily test this through the admin interface. You can find all the code for this demo in this GitHub repository.

Conclusion

Effective management of static and media files in Django projects is crucial for optimizing performance, security, and scalability. By utilizing tools like WhiteNoise for serving static files and AWS S3 for handling user-uploaded media, developers can streamline their applications and reduce server load. Additionally, implementing image optimization techniques, such as resizing and converting to WEBP format, can significantly enhance loading times and user experience. Adopting these best practices not only simplifies the deployment process but also ensures that applications are robust, efficient, and ready to handle high traffic demands.