Migrating Django App From Postgres to MongoDB

So, we've got Postgres, the SQL-compliant RDBMS, giving us that structured, organized feel. Then there's MongoDB, the NoSQL rebel, saying, "Who needs tables anyway? Not me!"

When you're trying to keep Jeff Bezos from raiding your piggy bank in AWS-RDS invoices, for your side projects (that, let's be honest, only a handful of people use), you start considering MongoDB.

Postgres and MySQL have been my go to for all things database-related. And SQLAlchemy? It's like the smooth operator of the ORM world, making queries feel like a breeze.

But NoSQL has its perks, especially when you're dealing with data that's as laid back as a cat on a sunny window sill. Enter MongoDB, where objects and documents can just chill without worrying about strict schemas.

Now, connecting MongoDB to Django or Flask? Pymongo is like the minimalistic approach – less fluff, more control. But then there are these other two layers of abstraction. It's like choosing between driving a stick shift or letting the autopilot do its thing.

And then there's Djongo. It lets me keep my Django models intact, saving me from the headache of restructuring everything. With Djongo, my app can smoothly transition to a MongoDB backend, and even our trusty ol' makemigrations and migrate commands don't break a sweat when the app.models decide to have a makeover.

So, you've birthed your Django app, you've got your models in order (we're skipping that drama). What's next? Let's dive into the MongoDB world with Djongo by our side, shall we?

Docker and Docker Compose Files

We want a mongoDB docker image running the app has access to a development DB for testing. You can checkout my blog post here covering this topic in fair details.

Create a docker-compose file like so:

version: "3.10"
services:

  app:
    build: .
    container_name: app-api
    ports:
      - "8000:8000"
    command: python /app/manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/app  
    depends_on:
      - mongo
      - db
    # mongo DB
  mongo:
    image: mongo:latest
    container_name: mongo
    environment:
      - MONGO_INITDB_ROOT_USERNAME=user
      - MONGO_INITDB_ROOT_PASSWORD=pass
      - MONGO_INITDB_DATABASE=mongo
      - MONGO_INITDB_USERNAME=user
      - MONGO_INITDB_PASSWORD=pass
    hostname: mongodb
    ports:
      - "27017:27017"

The Settings.py File

Make the following adjustments to the settings.py file:

DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'mongo', 
        'CLIENT': {
          'host': 'mongodb://mongodb:27017',
          'username': 'user',
          'password': 'pass',
          }
    }
}

This is simply telling Django we're using a mongo db database, with the djongo engine.

Requirements.py File

Django, Djongo integration is currently fraught with version compatibility issues, please maintain the versions in the requirement files as such:

django
pymongo==3.12.3
djongo==1.3.1
djangorestframework 
django-cors-headers

Models

The user models has to go through a some changes, because some of the Super Classes provided by Django User model will not work with the MongoDB backend, so we make adjustments.

We will create a custom user manager class that'll inherit from django.contrib.auth.models BaseUserManager and define a custom create_user and create_superuser methods.

from django.contrib.auth.models import AbstractBaseUser, BaseUserManager
from django.db import models
from django.contrib.auth.models import AbstractUser
from datetime import datetime  

class UserManager(BaseUserManager):
    def create_user(self, email, username, password=None, **extra_fields):
        if not email:
            raise ValueError('The Email field must be set')
        email = self.normalize_email(email)
        user = self.model(email=email, **extra_fields)
        user.set_password(password)
        user.save(using=self._db)
        return user

    def create_superuser(self, email, password=None, **extra_fields):
        extra_fields.setdefault('is_staff', True)
        extra_fields.setdefault('is_superuser', True)
        user = self.model(email=email, **extra_fields)
        user.set_password(password)
        user.save()
        return user

class User(AbstractBaseUser):
    email = models.EmailField(unique=True)
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)
    username = models.CharField(max_length=20, default=random.choice(default_usernames))
    is_active = models.BooleanField(default=True)
    is_staff = models.BooleanField(default=False)
    is_superuser = models.BooleanField(default=False)

    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = []
    objects = UserManager()

    def has_perm(self, perm, obj=None):
        return self.is_superuser

    def has_module_perms(self, app_label):
        return self.is_superuser

Lets also create a serializer.py file:

from rest_framework import serializers
from .models import *
from django.contrib.auth import get_user_model
from django.db import IntegrityError


class CreateUserSerializer(serializers.ModelSerializer):
    username = serializers.CharField()
    email = serializers.EmailField()
    password = serializers.CharField(write_only=True,
                                     style={'input_type': 'password'})
    class Meta:
        model = get_user_model()
        fields = ('username', 'password', 'email', 'first_name', 'last_name')
        write_only_fields = ('password')
        read_only_fields = ('is_staff', 'is_superuser', 'is_active')

    def validate(self, attrs):
        email = attrs.get('email', '')
        username = attrs.get('username', None)

        if User.objects.filter(email=email).exists():
            raise serializers.ValidationError(
                {'email': ('Email is already in use')})

        return super().validate(attrs)

    def create(self, validated_data):
        try:
            user = super(CreateUserSerializer, self).create(validated_data)
            user.set_password(validated_data['password'])
            user.save()
            return user
        except IntegrityError as ex:
            raise ValueError(ex)

class UserSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = ('id', 'email', 'first_name', 'last_name')

Though I hadn't mentioned this, but one can already see that what we're building is an user creation and authentication API endpoint using Django RestFramework DRF and therefore I opted for token based authentication.

Create User View

We can create a user registration view in view.py file:

from django.shortcuts import render
import datetime
from django.contrib.auth import authenticate, login, logout
from django.db import IntegrityError
from django.http import JsonResponse
from rest_framework.authtoken.views import ObtainAuthToken
from django.contrib.auth import get_user_model
from rest_framework.generics import CreateAPIView
from rest_framework.views import APIView
from rest_framework.permissions import AllowAny
from django.views import View
from rest_framework.authtoken.models import Token
from rest_framework.decorators import api_view, authentication_classes
from rest_framework.exceptions import ValidationError, APIException
from rest_framework.response import Response
from .models import *
from .serializers import *
import logging

class CreateUserAPIView(CreateAPIView):
    serializer_class = CreateUserSerializer
    permission_classes = [AllowAny]

    def create(self, request, *args, **kwargs):
        logging.info(f"CreateUserAPIView, create user request: {request}")
        serializer = self.get_serializer(data = request.data)
        serializer.is_valid(raise_exception=True)
        logging.info(f"CreateUserAPIView CreateRequest for user {request.data}")
        try:
            user = self.perform_create(serializer)
            headers = self.get_success_headers(serializer.data)

            logging.info(user)
            return Response(
                    {**serializer.data}
                )
        except ValueError as ex:
            logging.error(f"CreateUserAPIView ValueError: {ex} ")
            return JsonResponse({"error":str(ex)})
        except serializers.ValidationError as err:
            logging.error(f"CreateUserAPIView serializers.ValidationError: {err} ")   
            return JsonResponse({"error":str(err)})
        except Exception as ex:
            logging.exception(f"CreateUserAPIView Exception: {ex} ")

class UserAuth(ObtainAuthToken):
    """
    Serves request to authenticate a app user 
    sub-classing the ObtainAuthtoken rest-framework auth class
    """
    # serializer_class = UserSerializer
    permission_classes = [AllowAny]

    def post(self, request, *args, **kwargs):
        serializer = self.serializer_class(data=request.data,
                                           context={'request': request})

        try:
            serializer.is_valid(raise_exception=True)
            user = serializer.validated_data['user']
            logging.info(f"UserAuth Login request for user {user.email}")
            logging.info(f"UserAuth Login request for user {user.email}")

            token, created = Token.objects.get_or_create(user=user)
            user = UserSerializer(user)
            return Response({
                'token': token.key,
                'user': user.data
            })
        except ValidationError as err:
            logging.info(f"UserAuthr: {err} ")
            logging.error("UserAuth Error")
            return Response({"Error":str(err)})
        except Exception as ex:
            return JsonResponse({"Error": str(ex)}, safe=False, status=400)

The CreateUserAPIView class subclasses from he CreateAPIView provided in the rest_framework.generics module, and has a create method which is what it is.

class UserAuth(ObtainAuthToken) Serves request to authenticate a app user sub-classing the ObtainAuthtoken rest-framework auth class.

What they say about skinning cats, weird, there is more than one way an app might choose to authenticate users. Using the Django provided authenticate and login method. Use authenticate() to verify a set of credentials. It takes credentials as keyword arguments, username and password for the default case, checks them against each authentication backend, and returns a User object if the credentials are valid for a backend. If the credentials aren’t valid for any backend or if a backend raises PermissionDenied, it returns None. For example:

def login_view(request):
    if request.method == "POST":

        # Attempt to sign user in
        username = request.POST["username"]
        password = request.POST["password"]
        user = authenticate(request, username=username, password=password)

        # Check if authentication successful
        if user is not None:
            login(request, user)
            return HttpResponseRedirect(reverse("index"))
        else:
            return render(request, "app/login.html", {
                "message": "Invalid username and/or password."
            })
    else:
        return render(request, "app/login.html")

def register(request):
    if request.method == "POST":
        username = request.POST["username"]
        email = request.POST["email"]

        # Ensure password matches confirmation
        password = request.POST["password"]
        confirmation = request.POST["confirmation"]
        if password != confirmation:
            return render(request, "app/register.html", {
                "message": "Passwords must match."
            })

        # Attempt to create new user
        try:
            user = User.objects.create_user(username, email, password)
            user.save()
        except IntegrityError:
            return render(request, "app/register.html", {
                "message": "Username already taken."
            })
        login(request, user)
        return HttpResponseRedirect(reverse("index"))
    else:
        return render(request, "app/register.html")

This implies there has to be some basic html template files in place to redirect.

URLs

The urls.py file with the project folder should look like:

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include("app.urls")),
]

We can then create a urls.py file within the app folder:

from django.urls import path
from . import views

urlpatterns = [
    path("", views.index, name="index")
    # if you have a UserAuth view in place
    path("auth_login", views.UserAuth.as_view(), name="auth"),
    # or 
    # if you have the login_view view in place
    path("login", views.login_view, name="login"),
    # if you have a CreateUserAPIView view in place
    path("/create", views.CreateUserAPIView.as_view(), name="create"),
    # or 
    # if you have the register view in place
    path('register', views.register, name='register'),
    path("logout", views.logout_view, name="logout"),
]

I haven't created an index method in view, I assume you already have one, if one is not implemented you'll get an error.

If you went the route of using authenticate(), then you really don't need to follow the next steps. Your authentication system should be working as is.

Moving on, we need to edit the settings.py file, to tell Django we may want to authenticate user with RestFramework:

# add rest_framework and rest_framework.authtoken
INSTALLED_APPS = [
    'app',
    'django.contrib.admin',
    ...
    'rest_framework',
    'rest_framework.authtoken',
]

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'app.authenticate.BearerAuthentication',
        'rest_framework.authentication.TokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',

    ),
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.IsAuthenticated',
    ),

}

We've also added a few newlines here. The REST_FRAMEWORK switches for default auth class and permission classes.

In the DEFAULT_AUTHENTICATION_CLASSES I've added a custom class app.authenticate.BearerAuthentication because out of the box, DRF will issue a well, TokenAuthentication Token 'token string' , but I want to use BearerAuthentication instead.

BearerAuthentication

Create authentication.py with the app folder:

from rest_framework import authentication

class BearerAuthentication(authentication.TokenAuthentication):
    keyword = 'Bearer'

Everything is all setup. Running the command docker compose -f docker-compose.yml up --build creates 2 containers running on docker; app-api and mongo.

Migrate, Look Around 🌌

Time to play mad scientist with your container! In a sinister terminal ritual, summon the container spirits with docker exec -it app-api bash. Once inside, cast your spells – python manage.py makemigrations and python manage.py migrate. Watch as your database bows to your commands like a loyal minion.

Go ahead and create user by navigating to /register url on your browser or if you followed using DRF and have the postman app, send a POST request from postman to the /create.

You can look in your Mongo container to see collections that had been created by the migrations command. Run docker exec -it mongo bash . Mongo provides us with a command line interactive tool with mongosh , to interact with the mongo database using shell commands. Using the connection string mongodb://user:pass@mongo:27017, run mongosh "mongodb://user:pass@mongo:27017" --username=user . This logs the user into the default test db. To see all databases, run show dbs . Switch database with use <dbname> . To see all tables (read, collections), run show collections . You can view more mongo shell commands here.