Hello Techies👋! I’m Samiksha, Hope you all are doing amazing stuff. I’m back with another blog about Best MLOps practices for AI Engineering i.e. on Packaging your Code/Product as a Container (Creating Dockerfile). This Article covers absolute best practices for writing a Dockerfile with a Practical deployed application example “Advance AI Consultant Hybrid RAG Chatbot“. Please checkout the Code here: https://github.com/kolhesamiksha/Hybrid-Search-RAG.

Please NOTE: This blog is for beginners and professionals who wants to follow best Dockerfile writing practices to optimize your deployment time and reduce your Container Image size. If you wanted to know about RAG and building the RAG applications for your usecases checkout this blog: https://teckbakers.hashnode.dev/ai-consultant-hybrid-rag-chatbot.

Superb, Sounds Exciting!!🤩

Let’s dive into the Ocean with Docker Containers…🐳

As In this Article i have explained practical example from “Advance AI Consultant Hybrid RAG Chatbot“ usecase, Please check below image, which absolutely show the Built time reduction after following below Best practices for Dockerfile which will discuss further… Let’s go

What is Dockerfile? How Optimizing it affects the Application performance

A Dockerfile is a plain text document that contains a set of instructions used to build a Docker image. It acts as a blueprint for creating a containerized application, defining all the necessary components, configurations, and steps required to run the application consistently across different environments.

The Docker Image when combines with application libraries, settings, runtime environment becomes Docker Container.

In Docker, a Docker container is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, runtime, system tools, libraries, and settings. Containers are instances of Docker images, which act as templates or blueprints.

Dockerfile size reduction offers several benefits for containerized applications and development workflows:

Faster Builds and Deployments:

Smaller images require less time to download, transfer across networks, and load into the container runtime. This significantly speeds up build processes in CI/CD pipelines and accelerates deployment times, especially in environments with limited bandwidth or when deploying too many nodes.
Reduced Resource Consumption and Costs:

Leaner images consume less storage space on local machines, registries, and deployment environments (like Kubernetes clusters). This can lead to lower storage costs and more efficient utilization of resources.
Improved Security:

A smaller image footprint generally means fewer unnecessary components, libraries, and dependencies are included. This reduces the potential attack surface, as there are fewer possible vulnerabilities for attackers to exploit.
Faster Container Startup Times:

With less data to load and process, smaller images lead to quicker container startup times, which is crucial for applications requiring rapid scaling or in dynamic environments where containers are frequently started and stopped.
Simplified Maintenance and Debugging:

Smaller, more focused images are often easier to understand, maintain, and debug, as they contain only the essential components required for the application to run.
Enhanced Performance:

Overall application performance can benefit from smaller images due to faster loading and potentially reduced memory usage.

Now Let’s Refer Advance AI consultant Hybrid RAG Chatbot usecase Dockerfile to understand the Best practices one by one:

##################################### Stage 1: BUILD ######################################

# Use Python base image
FROM python:3.11-slim as builder

# Set working directory
WORKDIR /build

# Install system dependencies for Poetry
RUN apt-get update && apt-get install -y --no-install-recommends \
    apt-get install ffmpeg -y \
    make build-essential && \
    apt-get clean && rm -rf /var/lib/apt/lists/* && \
    pip install poetry

# Copy project files into the container
# hybrid_rag: SIZE: 0.19MB
COPY hybrid_rag hybrid_rag
COPY tests tests
COPY .pre-commit-config.yaml .pre-commit-config.yaml
COPY Makefile Makefile
COPY poetry.toml poetry.toml
COPY pyproject.toml pyproject.toml

# Optional: If No workflows/cicd setup for build test and deploy.. then use make install, make install-precommit, make run-precommit, make test, make clean commands
# to test your code locally.

# Build the wheel file using the Makefile
RUN make build

###################################### Stage 2: RUNTIME ###########################################
FROM python:3.11-slim

# Set working directory
WORKDIR /Hybrid-Search-RAG

# Install system dependencies for Poetry and Supervisor
RUN apt-get update && apt-get install -y --no-install-recommends \
    supervisor && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

COPY --from=builder /build/dist/*.whl .

# Install the wheel file && remove the wheel after installation, to save the memory space of virtual space of 
RUN pip install *.whl && rm -rf *.whl

##################
# chat_restapi -> SIZE: 0MB
# chat_streamlit_app -> SIZE: 0.23MB
##################

# Copy project files into the container
COPY chat_restapi chat_restapi
COPY chat_streamlit_app chat_streamlit_app
COPY .env.example chat_restapi/.env.example
COPY .env.example chat_streamlit_app/.env.example

#Supervisord.conf COPY
RUN mkdir -p /etc/supervisor/conf.d
COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf

# Expose ports for FastAPI and Streamlit
EXPOSE 8000 8501

# Run Supervisor to manage multiple processes (FastAPI + Streamlit)
CMD ["supervisord", "-n"]

Building the Layers - Explaination of Common Instructions

FROM: Docker starts by pulling the base image for the Container environment specified by the FROM instruction. If the image is not available locally, Docker pulls it from the Docker Hub or a private registry. lighter base images also helps reduce the Docker image size.
WORKDIR: Sets the working directory for subsequent instructions.
RUN: Executes commands in a new layer. For example, installing dependencies with RUN apt-get install -y curl.
COPY / ADD: Adds files or directories from the build context into the image. These files are copied into the image layer.
CMD / ENTRYPOINT: Specifies the command that should be executed when the container starts (if no command is provided by the user then runs the default command).
EXPOSE: Indicates which ports the container will listen on at runtime or the Container exposes to external IPs.
ENV: Sets environment variables inside the container.

I’ll discuss all of the Layers by considering above Dockerfile as reference with real-time scenario to better understand.

How Layers Work in Docker

Layered File System: Docker uses a layered file system, where each layer is read-only, and each new layer is built on top of the previous ones.
Caching: Docker caches each layer to speed up subsequent builds. If a layer’s contents don't change, Docker reuses that layer from cache, speeding up the build process.
Layering Strategy: Each RUN command in the Dockerfile results in a separate layer. The result of each layer is committed to the image as a new filesystem layer.
Docker uses caching to avoid rebuilding unchanged layers. However, when a new layer is created, it stores all the file changes, which can increase the image size.
For example, each RUN command may add a set of files (such as temporary files, cache, or logs) that are carried forward to the next layer

Let’s see RUN layer first, from the above Docker-Image adding the dependencies first in the Run Layer.. Adding single RUN command to download all dependencies also helps reduce the Image Size: Let’s see how below..

Why This is Done in a Single `RUN` Layer

Efficiency: By combining the package installation and cleanup commands into a single RUN step, you reduce the number of layers in the Docker image. Each RUN command creates a new layer, so minimizing the number of RUN commands helps reduce the final image size.
Image Size Optimization: The cleanup commands (apt-get clean and rm -rf /var/lib/apt/lists/*) remove unnecessary files right after the installation, ensuring that the Docker image is as small as possible.

Best Practices for Optimizing Docker Layers

Combine commands: Combine commands that can be logically grouped together into a single RUN statement to reduce the number of layers.
- Example: Installing dependencies and cleaning up in one RUN command.
Remove unnecessary files immediately: Remove temporary files, caches, and installation artifacts in the same RUN command where they were created.
- Example: Use apt-get clean and rm -rf /var/lib/apt/lists/* right after installing packages.
Order matters: Docker caches layers, so if a command changes, Docker has to rebuild all subsequent layers. For efficiency, put the most likely-to-change instructions (e.g., COPY or ADD of application files) near the end of the Dockerfile.

Increasing the number of RUN layers can increase the size of a Docker image because each layer carries with it the changes made in that step. Each new layer adds more metadata and can result in the accumulation of unnecessary files or temporary artifacts that aren't cleaned up, increasing the size of the final image. By combining RUN commands and removing unnecessary files in the same command, you can reduce the number of layers and optimize the image size.

What Exactly Happens when we run “Docker Build”?

The docker build command is used to create a Docker image from a Dockerfile. During this process, Docker reads the instructions in the Dockerfile and executes them step-by-step to build an image. Let’s break down the key stages of the docker build process to give you a detailed understanding of what happens behind the scenes:

1. Reading the Dockerfile

When you run the docker build command, Docker looks for a Dockerfile in the specified build context (typically the current directory, or a directory you specify).
The build context is the directory or folder that contains the files Docker needs to access (such as the Dockerfile, application files, dependencies, etc.).

2. Creating Build Context

Docker context is created and sent to the Docker daemon. This is the set of files (including the Dockerfile) that Docker will use to build the image.
The Docker daemon (the background service that manages Docker containers and images) receives this context, but only the files required for the build (files specified in the Dockerfile like COPY, ADD, etc.) are sent to the daemon.
Docker excludes files mentioned in the .dockerignore file, such as node_modules, .git, or logs, to reduce the context size.

How `.dockerignore` Works

Files and Directories to Ignore: The .dockerignore file lists patterns that match the files and directories to be excluded from the build context. The build context is the directory (and its contents) that gets sent to the Docker daemon during the docker build process.

3. Dockerfile Parsing

Docker parses the Dockerfile line by line, interpreting each instruction as a build step (e.g., RUN, COPY, ADD, EXPOSE, CMD, etc.).
Each instruction in the Dockerfile is executed in sequence to modify the image layer by layer.

4. Building Layers

Each instruction in the Dockerfile creates a new layer in the image.
- For example, a RUN command like RUN apt-get install -y curl will create a layer where the curl package is installed.
Layers are stored in a layered filesystem, which means that each new layer only contains the changes relative to the previous layer.
Docker uses caching for each layer: If a layer’s instruction hasn’t changed (i.e., the inputs are the same), Docker will reuse the existing layer from its cache instead of rebuilding it.
Layers are immutable once created, but Docker allows layer reuse to optimize subsequent builds (this speeds up the build process).

5. Optimizing with Caching

Docker checks whether any part of the build context or Dockerfile has changed since the last build. If it detects changes:
- It will re-execute the affected layers.
- If a layer hasn’t changed, Docker will use the cached version of that layer, which speeds up the build process significantly.
Example: If you modify a COPY instruction (or any line after it), Docker will rebuild that layer and all the layers after it, but layers before the change will be reused from cache.

6. Finalizing the Image

Once all instructions in the Dockerfile are processed, Docker will have a fully built image with all the layers.
Docker tags this image with a tag (e.g., myapp:latest), so it can be referred to and used later.
The image is stored in your local Docker registry, but it can also be pushed to a remote registry (e.g., Docker Hub, AWS ECR) for sharing or deployment.

Behind the Scenes: Step-by-Step Execution

Here’s a deeper look at what happens in a typical docker build process:

Preparation Phase:
- Docker prepares the context (excludes files in .dockerignore, compresses context if needed, etc.).
- Docker begins communicating with the Docker daemon.
Layer Creation Phase:
- For each instruction in the Dockerfile:
  - If the instruction has already been cached and the context hasn't changed, Docker reuses the cached layer.
  - If the context has changed or the layer doesn't exist, Docker creates a new layer by running the instruction.
  - Each instruction is executed in its own container, and the results are committed to a new image layer.
Image Layering:
- Each layer is independent, so Docker uses a copy-on-write filesystem (like AUFS or OverlayFS) to efficiently manage file changes between layers.
- When you modify a file in a later layer, the original layer is not modified; instead, the modified files are stored in the new layer.
Final Image:
- After all layers are built, Docker assembles the image and commits all layers into the final image, tagging it with the provided name (e.g., myapp:latest).
- Docker then stores the image in the local image registry and makes it available for running containers.

Now Let’s see this practically, By Run → Build → Push the above Dockerfile and see how this happens through logs -

Prerequisites: If you want to run this setup at your local Machine then first download Docker and then clone this repository: https://github.com/kolhesamiksha/Hybrid-Search-RAG then- run the below docker build -t rag . command from the root location where Dockerfile present.

As right now our Docker Image got created with a default name because we didn’t specified anything. Now first tag the image with desired name - NOTE: Below Image i’m pushing to AWS ECR so it required it’s name to be in a specified format.

If you wanted to know how to push the Local Docker Image to AWS ECR then follow this article where i discussed Deploy FastAPI using AWS ECR and AWS Lambda -

But without special specification, just by built, tag and push the image if you have Docker installed then it pushed the Docker Image to the Docker Desktop you installed.

You can see the image we pushed without any specific tag, got pushed to the Docker desktop and Now Docker Image got converted into Live Docker Container and Hence you code got packaged to run on any system.

And you can see the application i.e. FastAPI we created we can access using above container we just deployed...

Yeahh!🙌🏻!! Now we have covered everything to get you started to build and optimize the Dockerfile theoretically as well as practically for your Usecases and Entrerpise grade applications, So Your application will incure less infra cost…

Please feel free to contribute to this article in comments, share your insights and experience in optimizing docker images for your Large scale application. This will help everyone to learn from each others experience!!.

till then, Stay tuned and follow our newsletter to get daily updates & Built Project End to end!! Connect with me on linkedin, github, kaggle.

Let's Learn and grow together:) Stay Healthy stay Happy✨. Happy Learning!!

AI Engineering: Best Practices to Create a Dockerfile you ever know

Table of contents

What is Dockerfile? How Optimizing it affects the Application performance