How to efficiently Send Emails Asynchronously in Django

Sending email in a Django project is quite a simple task. You import the send_mail function and use it. We usually send emails at critical times, like when a visitor signs up, or for more specific tasks, like sending an autogenerated invoice or when something happens and you need to send a notification via email. When that email has to be sent from a request (HTTP), it can slow things down for several reasons. Maybe there are too many emails to be sent, or the mail server is taking time to respond, or something else, and then it becomes problematic due to errors like internal and other issues.

The immediate solution is to use something like Celery or Django RQ to send the email in the background. In this flow, you will probably write a task that takes some arguments such as the destination email, subject, message, and other things, and this task will then import send_mail to do the job. It’s great, but it has some limitations. First of all, this won’t really be possible if you are using some package that has the ability to send emails by themselves, for example, django-allauth when a user signs up or requests a password reset, and it’s far-fetched to decide to re-implement all that process yourself. At this point, the solution is to use an email backend that is designed to process email in the background by default. This way, you continue to write normal Django and everything works in the background, like django-celery-email or something else. This leads to another possible problem: it assumes I am always using an SMTP-based backend. I might be using a Postmark server token or SES API key or something else that is not native SMTP.

The solution is to write my own email backend that will get the email to be sent and call my own email backend the way it should be and send the mail asynchronously. This way, I don’t need to change the code already written. It can look something along these lines:

import logging
from postmarker.django import EmailBackend
from home.tasks import send_emails # << this task will send the mail in background
from home.utils import email_to_dict


logger = logging.getLogger(__name__)


class BackgroundEmailBackend(EmailBackend):
    """
    Load Postmark Email backend via a job and send mails in background
    """

    def __init__(self, fail_silently=False, **kwargs):
        super(BackgroundEmailBackend, self).__init__(fail_silently)
        self.init_kwargs = kwargs

    def send_messages(self, email_messages):
        logger.info("Sending email via RQ")
        send_emails.delay([email_to_dict(msg) for i, msg in enumerate(email_messages)], **self.init_kwargs)
        logger.info("Sending email Scheduled")

The content of home.utils.py

import logging
from django_rq import job
from django.conf import settings
from django.core.mail import get_connection
from home.utils import dict_to_email


logger = logging.getLogger(__name__)


@job
def send_emails(messages, **kwargs):
    # backward compat: handle **kwargs and missing backend_kwargs

    conn = get_connection(backend='postmarker.django.EmailBackend', **kwargs)
    try:
        conn.open()
    except Exception:
        logger.exception("Cannot reach POSTMARK_EMAIL_BACKEND %s", settings.CELERY_EMAIL_BACKEND)
    conn.send_messages([dict_to_email(msg_dict) for msg_dict in messages])
    conn.close()
    logger.info(f"Send {len(messages)} mails.")
    return messages

In this example, I am using Postmark to send emails, and I have installed the Django email backend for Postmark. This code is fairly customized yet simple and easy to change. You can even inherit from the regular Django email backend. I’ve taken inspiration from the django-celery library to write this, and I extracted a few methods from it, namely dict_to_email and email_to_dict. Their code is:

import copy
import base64
import time
from django.conf import settings
from django.template.loader import render_to_string
from django.utils.timezone import datetime, make_aware
from email.mime.base import MIMEBase
from django.core.mail import EmailMultiAlternatives, EmailMessage


def email_to_dict(message):
    if isinstance(message, dict):
        return message

    message_dict = {'subject': message.subject,
                    'body': message.body,
                    'from_email': message.from_email,
                    'to': message.to,
                    'bcc': message.bcc,
                    # ignore connection
                    'attachments': [],
                    'headers': message.extra_headers,
                    'cc': message.cc,
                    'reply_to': message.reply_to}

    if hasattr(message, 'alternatives'):
        message_dict['alternatives'] = message.alternatives
    if message.content_subtype != EmailMessage.content_subtype:
        message_dict["content_subtype"] = message.content_subtype
    if message.mixed_subtype != EmailMessage.mixed_subtype:
        message_dict["mixed_subtype"] = message.mixed_subtype

    attachments = message.attachments
    for attachment in attachments:
        if isinstance(attachment, MIMEBase):
            filename = attachment.get_filename('')
            binary_contents = attachment.get_payload(decode=True)
            mimetype = attachment.get_content_type()
        else:
            filename, binary_contents, mimetype = attachment
            # For a mimetype starting with text/, content is expected to be a string.
            if isinstance(binary_contents, str):
                binary_contents = binary_contents.encode()
        contents = base64.b64encode(binary_contents).decode('ascii')
        message_dict['attachments'].append((filename, contents, mimetype))


    return message_dict


def dict_to_email(messagedict):
    message_kwargs = copy.deepcopy(messagedict)  # prevents missing items on retry

    # remove items from message_kwargs until only valid EmailMessage/EmailMultiAlternatives kwargs are left
    # and save the removed items to be used as EmailMessage/EmailMultiAlternatives attributes later
    message_attributes = ['content_subtype', 'mixed_subtype']
    attributes_to_copy = {}
    for attr in message_attributes:
        if attr in message_kwargs:
            attributes_to_copy[attr] = message_kwargs.pop(attr)

    # remove attachments from message_kwargs then reinsert after base64 decoding
    attachments = message_kwargs.pop('attachments')
    message_kwargs['attachments'] = []
    for attachment in attachments:
        filename, contents, mimetype = attachment
        contents = base64.b64decode(contents.encode('ascii'))

        # For a mimetype starting with text/, content is expected to be a string.
        if mimetype and mimetype.startswith('text/'):
            contents = contents.decode()

        message_kwargs['attachments'].append((filename, contents, mimetype))

    if 'alternatives' in message_kwargs:
        message = EmailMultiAlternatives(**message_kwargs)
    else:
        message = EmailMessage(**message_kwargs)

    # set attributes on message with items removed from message_kwargs earlier
    for attr, val in attributes_to_copy.items():
        setattr(message, attr, val)

    return message

These functions are important because most of the time when you send an object to a background processing tool, you need to serialize it and deserialize it when reading it from the workers (the process that runs in the background to send the actual emails). It’s better to send native Python objects. In our case, we are sending an EmailMessage instance, so we use email_to_dict when sending the email to the background job and dict_to_mail when reading it from the queue in the background worker. You can learn more about the original version here from the django-celery-email package: https://github.com/pmclanahan/django-celery-email/blob/d47da19c09e29eea90684692e8dfa059e026c046/djcelery_email/utils.py#L26. I had to perform some small tweaks and remove some code since I am using django-rq to handle background tasks. But this method is valid regardless of the background processing you are using.

While writing this article, I discovered the package django-mailer, which does this but better because it actually lets you specify whatever email backend you want to use. Hence, you have the same outcome with more flexibility: check its usage guide here https://github.com/pinax/django-mailer/blob/master/docs/usage.rst#usage.

I build projects using Django and Wagtail CMS. If you're planning to start your next tech venture, feel free to reach out. I can help you make the right decisions, even if we don't end up working together. Just message me on X (Twitter) @adonis__simo.