Docker

Arindam BaidyaArindam Baidya
29 min read

Table of contents

Docker Overview

Why do we need docker ?

Docker is a tool that lets you package your application with everything it needs (code, libraries, settings) into a single unit called a container, so it can run anywhere โ€” on any computer, server, or cloud โ€” exactly the same way.


What it can do?

Docker is a tool that lets you run applications in small, lightweight packages called containers, so they work the same on any computer or server.


What are container?

Containers are lightweight, standalone, and executable software packages that include everything needed to run a piece of software, including the code, runtime, system tools, libraries, and settings. They are isolated from each other and the host system, ensuring that they run consistently across different computing environments. Containers are a core component of Docker, allowing applications to be deployed and run reliably regardless of the underlying infrastructure.


Virtual machine vs Container

๐Ÿ” Feature๐Ÿ–ฅ๏ธ Virtual Machine (VM)๐Ÿณ Container
๐Ÿ’ก What it isA full computer inside your computer ๐Ÿ A lightweight app package ๐Ÿ“ฆ
๐Ÿง  OS Included?Has its own full operating system ๐ŸงShares the host OS kernel ๐Ÿ‘ฅ
โš™๏ธ SpeedSlower to start ๐Ÿข (minutes)Very fast ๐Ÿš€ (starts in seconds)
๐Ÿงฑ SizeLarge file size ๐Ÿ“‚ (GBs)Small and lightweight ๐Ÿ’พ (MBs)
๐Ÿ”„ PortabilityHarder to move or copy ๐Ÿ”—Easy to share or move ๐ŸŒ
๐Ÿ” IsolationFully isolated, like separate houses ๐Ÿ˜๏ธIsolated, but share some things, like rooms in a house ๐Ÿก
๐Ÿ“ค DeploymentNot ideal for frequent updates ๐Ÿ”„Great for CI/CD & microservices ๐Ÿ”
๐Ÿงฐ Resource UsageUses more CPU and RAM ๐Ÿ’ฅVery efficient ๐Ÿง 
๐Ÿ› ๏ธ Setup TimeLonger setup โณQuick setup โšก
๐Ÿ”ง ExampleWindows running inside Linux using VirtualBox ๐ŸชŸA Node.js app running in a Docker container ๐ŸŒ

๐ŸŒ What is a Public Docker Registry?

A public Docker registry is an online library where developers can store, share, and download Docker containers (called images).


๐Ÿณ What is Docker Hub?

Docker Hub is the official public Docker registry provided by Docker.

Itโ€™s like the Play Store or App Store, but for Docker images!


๐Ÿงฑ Docker๐ŸŒ Docker Hub
A tool to create and run containersA website to store and share container images
You build images using DockerYou upload/download those images from Docker Hub
You run docker pull or docker pushThese commands talk to Docker Hub

๐Ÿณ Container vs ๐Ÿ–ผ๏ธ Image (with Examples)

๐Ÿ” Feature๐Ÿ–ผ๏ธ Docker Image๐Ÿ“ฆ Docker Container
๐Ÿ’ก What is it?A blueprint or template for your app ๐Ÿ“A running instance of that blueprint ๐Ÿš€
๐Ÿ“ฆ Think of it likeA recipe for a cake ๐ŸฐThe actual cake made from the recipe ๐ŸŽ‚
โฑ๏ธ StateStatic (doesnโ€™t change) ๐ŸงŠDynamic (runs and can change) ๐Ÿ”„
๐Ÿƒ Runs or not?Cannot run on its own ๐Ÿ›‘Can run as a process ๐ŸŸข
๐Ÿง  Use caseUsed to create containersUsed to run the actual app
๐Ÿ› ๏ธ Created withdocker builddocker run
๐Ÿ—ƒ๏ธ Stored inDocker Hub or local storage ๐Ÿ“Runs in your systemโ€™s memory ๐Ÿ”ง
๐Ÿ”„ Changes allowed?Cannot change directly โŒYou can make changes while it runs โœ…
๐Ÿ’ฌ Example commanddocker pull nginx โ†’ gets imagedocker run nginx โ†’ starts the container

Basic Docker Commands

โ–ถ๏ธ 1. Running Containers

CommandDescription
docker run <image>Runs a container from an image in foreground mode
docker run -d <image>Runs the container in detached/background mode
docker run ubuntuStarts Ubuntu container, but exits immediately (no process)
docker run ubuntu sleep 5Runs Ubuntu and executes the command sleep 5
docker run -it ubuntuRuns Ubuntu in interactive mode with terminal access
docker run -it ubuntu bashStarts an interactive Ubuntu shell using bash

๐Ÿ“‹ 2. Viewing Containers

CommandDescription
docker psLists only running containers
docker ps -aLists all containers (running, stopped, exited)

โน๏ธ 3. Stopping and Removing Containers

CommandDescription
docker stop <container_id/name>Stops a running container
docker rm <container_id/name>Removes a stopped container permanently

๐Ÿ–ผ๏ธ 4. Working with Images

CommandDescription
docker imagesLists all downloaded images with their size
docker pull <image>Downloads an image from Docker Hub (does not run it)
docker rmi <image>Deletes an image (must remove its containers first)

โš™๏ธ 5. Running Commands Inside Containers

CommandDescription
docker exec <container_name> <command>Runs a command inside a running container
Example: docker exec distracted_mcclintock cat /etc/hostsReads a file from inside the container

๐Ÿ”€ 6. Attach / Detach Mode

CommandDescription
docker run kodekloud/simple-webappRuns the container in attached mode (foreground)
docker run -d kodekloud/simple-webappRuns in detached/background mode
docker attach <container_id>Reattaches to a running container in foreground mode

โ†’ Run the command docker version and look for the version of Client and Server Engine.

โ†’ Run a container with the nginx:1.14-alpine image and name it webapp

  • Run the command docker run -d --name webapp nginx:1.14-alpine and check the status of created container by docker ps command.

โ†’ Delete all images on the host

  • Stop and delete all the containers being used by images.

  • Then run the command to delete all the available images: docker rmi $(docker images -aq)

โ†’ docker run redis:4.0 here the version (4.0) specified concider as TAG. When we donโ€™t specify for tag, docker consider it as latest. We can find all the versions on https://hub.docker.com/


โœ… Goal:

We will create a simple Python application that:

  • Asks the user for their name via standard input (input()).

  • Greets them with a welcome message: Welcome, <name>!.

We'll then run it using Docker in both non-interactive and interactive (-it) modes to demonstrate the difference.


โœ… Step 1: Create the Python App

app.py

name = input("Enter your name: ")
print(f"Welcome, {name}!")

โœ… Step 2: Create a Dockerfile

Dockerfile

FROM python:3.11-slim

WORKDIR /app
COPY app.py .

CMD ["python", "app.py"]

โœ… Step 3: Build the Docker Image

docker build -t python-input-demo .

โœ… Step 4: Run the Container (Non-Interactive Mode)

docker run python-input-demo

Output:

Enter your name:

โžก๏ธ It hangs or exits immediately without taking input.


โœ… Step 5: Run the Container with -it (Interactive Mode)

docker run -it python-input-demo

Output:

Enter your name: Arindam
Welcome, Arindam!

โœ… Success! Because -it attaches a pseudo-TTY and lets you interact with the container's STDIN.

๐Ÿณ Docker Port Mapping


๐Ÿง  What Is Port Mapping?

Docker containers run in isolated environments. If a container serves content on port 80, you canโ€™t access it directly from your computer unless you map that port to your host machine.

๐Ÿ‘‰ Port Mapping connects:

<your-host-machine>:<host-port> โž <container>:<container-port>

๐Ÿ“ฆ Example App: Nginx Web Server

Nginx is a lightweight web server that listens on port 80 by default.


โœ… Option 1: Run Container With Port Mapping

๐Ÿ”น Command:

docker run -d --name mynginx -p 8080:80 nginx

๐Ÿ”น What it does:

  • Runs Nginx in the background (-d)

  • Maps your hostโ€™s port 8080 โž containerโ€™s port 80

  • Container is named mynginx

๐Ÿ”น Test it:

Open your browser or run:

curl http://localhost:8080

โœ… Youโ€™ll see the default Nginx page.


๐Ÿšซ Option 2: Run Container Without Port Mapping

๐Ÿ”น Command:

docker run -d --name mynginx2 nginx

This runs Nginx without mapping its port to the host, so:

  • You cannot access it via localhost

  • You must access it via the containerโ€™s internal IP


๐Ÿ” How to Access the App via Container IP (No Port Mapping)

Step 1: Find the container IP:

docker exec <container_name_or_id> hostname -i

๐Ÿ”น Output:

172.17.0.3

Thatโ€™s the internal Docker IP.

Step 2: Access the container (only works from the host, not the internet):

curl http://172.17.0.3

โœ… Youโ€™ll see the Nginx welcome page even without port mapping.

๐Ÿงจ This wonโ€™t work from another device unless you do port mapping.


๐Ÿ“ฆ Docker Volume Mapping โ€“ MySQL Example


๐Ÿง  Why Use Volumes?

By default, when you remove a container, all data stored inside it is lost. Volumes allow you to:

  • Persist database data

  • Share files between host and container

  • Backup and restore easily


๐Ÿ”ด Scenario 1: Running MySQL Without Volume Mapping

docker run --name mysql -e MYSQL_ROOT_PASSWORD=root -d mysql
  • Starts a MySQL container.

  • Password is set to root.

  • No volume is mounted.

โŒ Problem:

docker stop mysql
docker rm mysql

Now all data (e.g., tables, users, etc.) is lost forever, because /var/lib/mysql was inside the container.


โœ… Scenario 2: Run MySQL With Volume Mapping

๐Ÿ”น Step 1: Run MySQL with a volume:

docker run --name mysql -e MYSQL_ROOT_PASSWORD=root \
-v /opt/datadir:/var/lib/mysql \
-d mysql
  • -v /opt/datadir:/var/lib/mysql maps:

    • /opt/datadir on the host

    • to /var/lib/mysql in the container (where MySQL stores its data)

๐Ÿ”น Step 2: Stop and Remove Container

docker stop mysql
docker rm mysql

๐Ÿšซ Container is gone, but...

๐Ÿ”น Step 3: Data is still safe on the host:

ls /opt/datadir

๐Ÿ“ Youโ€™ll see MySQLโ€™s database files.


๐Ÿ”น Step 4: Start a New Container with Same Volume

docker run --name mysql2 -e MYSQL_ROOT_PASSWORD=root \
-v /opt/datadir:/var/lib/mysql \
-d mysql

โœ… MySQL will reuse the existing data from /opt/datadir.


๐Ÿ› ๏ธ Common Docker Inspection & Logging Commands

CommandWhat It Does
docker inspect <name>Shows detailed container info (JSON)
docker logs <name>Shows container stdout/stderr logs
docker logs -f <name>Follows logs live (real-time tail)

Docker Images

How to create an image file ?

To create a Docker image file for your Flask web application, follow these steps:

  1. Create a Dockerfile: This file contains the instructions to build the Docker image.

     FROM ubuntu
     RUN apt-get update
     RUN apt-get install -y python3 python3-pip
     RUN pip3 install flask flask-mysql
     COPY . /opt/source-code
     ENTRYPOINT FLASK_APP=/opt/source-code/app.py flask run
    
  2. Build the Docker Image: Use the Dockerfile to build your image and tag it appropriately.

     docker build -t arindam/my-custom-app .
    
  3. Push to Docker Hub: If you want to make the image available publicly, push it to Docker Hub.

     docker push arindam/my-custom-app
    

By following these steps, you will have created a Docker image for your Flask web application, which can be shared and deployed across different environments.


๐Ÿงฑ Layered Architecture in Docker (Explained with Dockerfile)

When you build a Docker image, each line in your Dockerfile creates a new layer in the image. These layers are stacked on top of each other to form the final image.


๐Ÿ” What is a Layer?

  • A layer is a read-only snapshot of the file system at a certain point during image build.

  • Docker caches layers and reuses them when possible to speed up builds.

  • Only the top layer (created when a container is run) is writable.


๐Ÿ“„ Example Dockerfile (Layered Breakdown)

DockerfileCopyEditFROM ubuntu:20.04                        # Layer 1
RUN apt-get update -y                   # Layer 2
RUN apt-get install -y python3 pip      # Layer 3
COPY . /opt/source-code                 # Layer 4
WORKDIR /opt/source-code                # Layer 5
RUN pip install -r requirements.txt     # Layer 6
ENV FLASK_APP=app.py                    # Layer 7
ENTRYPOINT ["flask", "run", "--host=0.0.0.0"]  # Layer 8

docker history <image-name> shows a layer-by-layer breakdown of how a Docker image was built, including commands used and layer sizes.


๐ŸŽจ Using Environment Variables in Applications & Docker

๐Ÿง  Application Code with Hardcoded Value:

color = "red"

โœ… Step 1: Replace with Environment Variable

Update your code to:

import os

color = os.environ.get('APP_COLOR', 'red')  # Default: red

๐Ÿš€ Step 2: Run with a Custom Value in Terminal

export APP_COLOR=blue
python app.py

โžก๏ธ This sets color = "blue" at runtime!


๐Ÿณ Using ENV Variables in Docker

๐Ÿ”น Run Docker Container with ENV Variable:

docker run -e APP_COLOR=blue simple-webapp-color

๐Ÿ” Inspect ENV Variable Inside a Running Container:

docker inspect <container_name> | grep APP_COLOR

Or see all environment variables:

docker inspect <container_name> | grep -i env

๐Ÿณ Docker: CMD vs ENTRYPOINT โ€“ What's the Difference?


๐Ÿ”น ENTRYPOINT โ€“ The Main Command

Think of this as the โ€œalways run this commandโ€ part of your container.

โœ… Fixed command
โœ… Hard to override (unless you use --entrypoint)
โœ… Great for containers that always run one app (e.g., nginx, python, java)

Example:

ENTRYPOINT ["python"]
CMD ["app.py"]

๐Ÿ”ธ On container start โ†’

python app.py

๐Ÿ”ธ If you run:

docker run myimage script.py

It runs:

python script.py

๐Ÿ”น CMD โ€“ The Default Arguments

This is the "default instruction" when no command is passed.

โœ… Can be overridden easily at runtime
โœ… Works with or without ENTRYPOINT
โœ… Good for setting default parameters

Example:

CMD ["echo", "Hello from CMD"]

๐Ÿ”ธ On container start โ†’

echo Hello from CMD

๐Ÿ”ธ But if you run:

docker run myimage echo "Hi there"

โœ… It overrides CMD!


๐Ÿ”„ Combined Usage

ENTRYPOINT ["python"]
CMD ["app.py"]
  • ๐ŸŸข docker run myimage โ†’ python app.py

  • ๐ŸŸก docker run myimage test.py โ†’ python test.py


Docker Compose

๐Ÿณ Setting Up Multi-Service Applications with Docker Compose

When setting up a complex application that runs multiple services, Docker Compose is the better approach. Instead of running several docker run commands manually, you can define your entire application stack in a single YAML configuration file named docker-compose.yaml.

With docker-compose, you:

โœ… Define all services and their configurations
โœ… Run the complete stack using docker-compose up
โœ… Maintain a version-controlled configuration in one file
โ— Note: Docker Compose is intended for a single Docker host (not multiple nodes)


๐Ÿ—ณ๏ธ Example: Simple Voting Application

This is a basic voting application stack with the following components:

ComponentDescription
voteA Python web app to vote for Cats or Dogs
redisAn in-memory database that stores each vote
workerA .NET app that processes the votes and updates PostgreSQL
db (Postgres)Stores total counts of votes
resultA Node.js app to display the vote results

๐Ÿงฑ Application Architecture:

User
 โ†“
Vote App (Python) โ†’ Redis
                     โ†“
                Worker (.NET) โ†’ PostgreSQL
                                         โ†“
                               Result App (Node.js)

๐Ÿ”ง Running Services Individually (Using docker run)

Assuming all images are pre-built and available:

# Run Redis
docker run -d --name=redis redis

# Run PostgreSQL
docker run -d --name=db postgres:9.4

# Run Vote app with Redis link
docker run -d --name=vote -p 5000:80 --link redis:redis voting-app

# Run Result app with DB link
docker run -d --name=result -p 5001:80 --link db:db result-app

# Run Worker with links to both Redis and DB
docker run -d --name=worker --link redis:redis --link db:db worker

๐Ÿ”— --link creates aliases and adds entries to /etc/hosts, allowing containers to resolve each other by name.


โœ… Docker Compose!

Instead of running each container manually, define all of them in a docker-compose.yaml file:

version: '3'

services:
  redis:
    image: redis
    container_name: redis

  db:
    image: postgres:9.4
    container_name: db

  vote:
    image: voting-app
    container_name: vote
    ports:
      - "5000:80"
    depends_on:
      - redis

  result:
    image: result-app
    container_name: result
    ports:
      - "5001:80"
    depends_on:
      - db

  worker:
    image: worker
    container_name: worker
    depends_on:
      - redis
      - db

๐Ÿ’ก depends_on ensures that services are started in the right order.


๐Ÿš€ Launching the Stack

Once the docker-compose.yaml is ready, you can bring up the entire stack with:

docker-compose up -d

To stop and remove everything:

docker-compose down

๐Ÿ”ง Docker Engine

  • Definition: The core part of Docker, allowing building, running, and managing containers.

  • Architecture:

    • docker (CLI) โ€”> sends commands via REST API โ€”> dockerd (daemon)
  • Components:

    • Docker Daemon (dockerd)

    • Docker Client (docker)

    • Docker REST API

  • Edit Docker Engine Host:

      // /etc/docker/daemon.json
      {
        "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2375"]
      }
    
      sudo systemctl restart docker
    

๐ŸŒ Remote Docker Access

  • Insecure (no TLS):

      docker -H tcp://<REMOTE_IP>:2375 ps
    
  • Secure (with TLS):

      docker --tlsverify \
        --tlscacert=ca.pem \
        --tlscert=cert.pem \
        --tlskey=key.pem \
        -H=tcp://<REMOTE_IP>:2376 ps
    

๐Ÿงฑ Namespaces in Docker

Namespaces isolate resources to create container environments.

NamespaceIsolatesPurpose
PIDProcess IDsOnly see containerโ€™s own processes
NETNetwork interfacesIndependent network stack
MNTMount pointsIsolated filesystem
UTSHostname/domain nameSeparate hostname
IPCInter-process communicationSeparate shared memory etc.
USERUID/GIDMaps user IDs

๐Ÿ“Œ Used to make containers feel like they are separate systems.


๐Ÿงฎ cgroups (Control Groups)

  • Purpose: Limit, monitor, and isolate resource usage of processes.

  • Docker uses cgroups to control:

    • CPU

    • Memory

    • Disk I/O

    • Number of processes (PIDs)

    • Devices

  • Example Command:

      docker run --memory="256m" --cpus="1" nginx
    
  • Monitor container usage:

      docker stats <container_id>
    
  • cgroup location in Linux:

      /sys/fs/cgroup/
    

๐Ÿณ Docker Storage


๐Ÿ“ File System Structure in Docker

When Docker is installed on a system, it creates a specific folder structure at:

/var/lib/docker

This directory contains several subdirectories:

FolderPurpose
aufs/ or overlay2/Stores layered file system data used by containers
containers/Stores logs and metadata of individual containers
volumes/Stores data volumes for persistent storage
image/Stores all image-related data

๐Ÿ”น Important: Docker stores images, containers, volumes, and other metadata in this location by default.


๐Ÿงฑ Layered Architecture of Docker Images

Docker builds images using a layered architecture. Each line in the Dockerfile forms a new layer, which contains only the changes made by that instruction.

โœ… Example:

  1. FROM ubuntu โ†’ Base image layer

  2. RUN apt-get install โ†’ Adds system packages

  3. RUN pip install flask โ†’ Adds Python packages

  4. COPY . /app โ†’ Adds source code

  5. ENTRYPOINT ["python", "app.py"] โ†’ Defines startup command

๐Ÿ”„ Layer Reuse (Cache Efficiency)

When building another application with the same base image and dependencies, Docker reuses the common layers from cache, creating only the new layers (e.g., source code and entrypoint).

โœ… This approach:

  • Saves build time

  • Reduces disk space usage


๐Ÿง  Understanding the Copy-on-Write (CoW) Mechanism

Image layers are read-only.

๐Ÿ”ง What happens when a file is modified inside a container?

  1. Docker copies the original file to the containerโ€™s read-write layer.

  2. All future modifications happen on this copied version.

  3. This is called the Copy-on-Write mechanism.

โœ… Benefits:

  • Keeps image layer unchanged and reusable

  • Allows multiple containers to share the same image layer


๐Ÿ“ฆ Container Writable Layer

When a container is created (docker run), Docker adds a writable layer above the image layers.

This layer stores:

  • Logs

  • Temporary files

  • Any data written by applications

โ— This writable layer is:

  • Ephemeral โ€” it exists only as long as the container exists.

  • Deleted when the container is removed.


๐Ÿ’พ Persisting Data Using Volumes

๐Ÿ”น Problem: What if you want to preserve data (e.g., DB records) even after container deletion?

โœ… Solution: Use Volumes


๐Ÿ—‚๏ธ Volume Mounting (Managed by Docker)

Step 1: Create a volume

docker volume create data_volume

This creates a directory under:

/var/lib/docker/volumes/data_volume/_data

Step 2: Run container with volume

docker run -v data_volume:/var/lib/mysql mysql
  • data_volume is mounted to /var/lib/mysql inside the container.

  • All DB writes go to this volume on the host.

  • Data persists even after the container is deleted.

๐Ÿ’ก If you skip creating the volume beforehand:

docker run -v data_volume2:/var/lib/mysql mysql

Docker will auto-create the data_volume2.


๐Ÿ“‚ Bind Mounting (Using Host Directory)

If you already have data on the host (e.g., /data/mysql) and want to use that:

docker run -v /data/mysql:/var/lib/mysql mysql

This maps the host folder /data/mysql directly into the container.

๐Ÿ”„ This is called Bind Mounting.


๐Ÿ†• Modern Way: Using --mount Option

The -v flag is considered the old style.

๐Ÿงพ Preferred: --mount

docker run \
  --mount type=bind,source=/data/mysql,target=/var/lib/mysql \
  mysql

Key-Value Options:

OptionValue
typebind or volume
sourcePath on the host system
targetMount point inside the container

โš™๏ธ Storage Drivers โ€“ The Core Behind Docker Storage

Docker uses Storage Drivers to manage:

  • Layered image architecture

  • Copy-on-write functionality

  • Container writable layer

Common Storage Drivers:

DriverNotes
AUFSDefault on Ubuntu systems
Overlay2Modern driver, widely used across Linux distros
Device MapperGood support on CentOS and Fedora
BTRFSAdvanced features, less commonly used
ZFSEnterprise-grade features

โœ… Docker auto-selects the best storage driver depending on the host OS.


๐ŸŒ Docker Networking


๐Ÿš€ Overview

When Docker is installed, it automatically creates 3 default networks:

Network TypeDescription
bridgeDefault network for containers
noneIsolates container from all networks
hostShares hostโ€™s network directly

๐Ÿงฐ Basic Networking Commands

PurposeCommand Example
Run container in default bridgedocker run ubuntu
Run container in none networkdocker run ubuntu --network=none
Run container in host networkdocker run ubuntu --network=host

๐Ÿ› ๏ธ Bridge Network (Default)

  • A private internal network created by Docker on the host.

  • Containers get an internal IP (typically from 172.17.x.x range).

  • Containers can communicate with each other using internal IPs.

  • External access requires port mapping:

      docker run -p 8080:5000 webapp
    

๐Ÿ” Example:

docker run -d --name web1 nginx
docker run -d --name web2 nginx
# web1 and web2 can talk to each other using 172.17.x.x or by container name

๐Ÿ  Host Network

  • The container shares the same network interface as the host.

  • No network isolation โ†’ directly uses hostโ€™s IP and ports.

  • Useful for performance or access simplicity (e.g., for servers).

โœ… Pros:

  • No need for port mapping (-p flag not required).

  • Fast network performance.

โš ๏ธ Cons:

  • Port conflicts: Canโ€™t run multiple containers on the same port.

  • Breaks isolation between host and container.


๐Ÿšซ None Network

  • The container is not attached to any network.

  • No external access.

  • No communication with other containers.

  • Best for security and sandboxing.


๐Ÿ”ง Creating a Custom Bridge Network

If you want containers to be grouped or isolated, create a custom internal network:

docker network create \
--driver bridge \
--subnet 182.18.0.0/16 \
custom-isolated-network

๐ŸŽฏ Benefits:

  • Group containers logically.

  • Provide isolation between groups.

  • Assign static subnets.


๐Ÿ“œ Listing & Inspecting Networks

TaskCommand
List all networksdocker network ls
Inspect network settingsdocker inspect <container_name>
Check containerโ€™s IP, MACSee NetworkSettings section

๐Ÿ” Container-to-Container Communication

๐Ÿงญ Access by IP

  • You can use a containerโ€™s internal IP like 172.17.0.3.

  • โŒ Not reliable: IPs may change after restarts.

๐Ÿงฑ Access by Name โœ… (Best Practice)

  • Containers can resolve each other by name.

  • Docker runs a built-in DNS server at 127.0.0.11 inside containers.

๐Ÿ”น Example:

  • Web app container can connect to MySQL using:

      mysql://mysql_container:3306
    
  • As long as both are on the same Docker network.


๐Ÿงช Under the Hood: How Docker Implements Networking

1. Network Namespaces

  • Each container runs in its own network namespace.

  • Provides complete isolation: interfaces, routing tables, etc.

2. Virtual Ethernet (veth) Pairs

  • Docker connects containers using veth pairs:

    • One end inside container, other end in bridge.

    • Acts like a virtual network cable.

3. Bridge Interface

  • Docker creates a bridge (e.g., docker0) on the host.

  • All container interfaces connect to this bridge.

4. Built-in DNS Server

  • Docker includes an embedded DNS server to resolve container names.

  • Internal IP: 127.0.0.11 in containers.

  • Automatically updates when containers join/leave a network.


Docker Registry

๐Ÿ“ฆ What is a Docker Registry?

A Docker Registry is a storage and distribution system for Docker images. It is where Docker images are stored, managed, shared, and retrieved.


๐Ÿงพ Key Concepts

TermDescription
Docker ImageA packaged application including code, libraries, environment settings
RegistryA system that stores Docker images and makes them available for download
RepositoryA collection of related images (usually different versions of the same app)
TagLabel used to identify image versions (e.g., latest, v1.0, etc.)

๐ŸŒ Types of Docker Registries

1. Docker Hub (Public Cloud Registry)

  • Default registry used by Docker.

  • Hosted at: https://hub.docker.com

  • You can pull images directly:

      docker pull nginx
    
  • Or push your own:

      docker push username/myapp
    

2. Private Registry

  • You can host your own Docker registry within your organization:

      docker run -d -p 5000:5000 --name registry registry:2
    
  • Useful for:

    • Internal development

    • Restricted access

    • Faster access inside local network

3. Other Registries

  • Amazon ECR (Elastic Container Registry)

  • Google Container Registry (GCR)

  • GitHub Container Registry (GHCR)

  • Harbor (open-source enterprise-grade registry)


โš™๏ธ Docker Registry vs Repository

  • Registry is the server that stores images.

  • Repository is a collection of images (usually different versions of the same app) in the registry.

๐Ÿ“Œ Example:

  • nginx โ†’ a repository

  • nginx:latest, nginx:1.21 โ†’ image tags in that repository

  • Stored at Docker Hub, which is the registry


๐Ÿ” Common Docker Registry Commands

ActionCommand Example
Pull an imagedocker pull ubuntu
Push an imagedocker push myrepo/myimage:tag
Login to registrydocker login
Tag local imagedocker tag image myrepo/image:tag
Run private registrydocker run -d -p 5000:5000 registry:2

๐Ÿ›ก๏ธ Authentication & Access Control

  • Docker Hub supports public and private repositories.

  • Private registries can be secured using:

    • HTTPS

    • Authentication mechanisms

    • Role-based access control

StepActionExample Command
1Login to registrydocker login myregistry.com
2Pull imagedocker pull nginx
3Tag for private registrydocker tag nginx myregistry.com/myuser/nginx:v1
4Push to registrydocker push myregistry.com/myuser/nginx:v1
5View image on registrycurl http://myregistry.com/v2/_catalog
6Pull from registrydocker pull myregistry.com/myuser/nginx:v1
7Run container from registrydocker run -d -p 8080:80 myregistry.com/myuser/nginx:v1
8Stop and remove containerdocker stop nginx-test && docker rm nginx-test
9Remove unused imagesdocker image prune -a

โš™๏ธ Container Orchestration โ€“ Explained Visually


๐Ÿณ Docker and Its Limitation

๐Ÿ”น Running a Single Application

You can run a Node.js application with Docker using:

docker run nodejs
  • โœ… Simple and fast for local development

  • โŒ But... it only runs one instance on one host


โŒ The Problems with Manual Scaling

ChallengeDescription
Manual scalingYou must run docker run multiple times to scale the app
Manual monitoringYou must monitor performance and container health yourself
No fault recoveryIf a container fails, you must restart it manually
Single host dependencyIf the Docker host crashes, all containers go down

๐Ÿš€ Solution: Container Orchestration

A system for managing, scaling, healing, and automating containerized applications across multiple Docker hosts.


๐Ÿงฐ What It Does

  • ๐Ÿ”„ Automatic deployment of containers

  • ๐Ÿ“ˆ Auto-scaling: scale up when demand increases

  • ๐Ÿ“‰ Auto down-scaling: scale down when demand drops

  • โ™ป๏ธ Self-healing: restart failed containers

  • ๐Ÿงญ Load balancing: distribute traffic across instances

  • ๐ŸŒ Multi-host support: containers can run across many nodes

  • ๐Ÿ” High availability: no single point of failure


๐Ÿ–ฅ๏ธ Typical Setup

[ Orchestration System ]
        |
+----------------------+
|    Multiple Hosts    |
+----------------------+
| Host 1 | Host 2 | ...|
| Nodejs | Nodejs | ...|
+----------------------+

Each host runs Docker and is controlled by the orchestration tool.


๐Ÿงช Docker Swarm Example

docker service create --replicas=100 nodejs
  • ๐ŸŽฏ Deploys 100 instances of the nodejs application

  • ๐Ÿ“ก Swarm manages distribution, health, and load balancing


ToolHighlights
Docker SwarmNative Docker tool, simple setup, CLI integrated
KubernetesMost popular, powerful, complex, used at scale
NomadLightweight, by HashiCorp, easy integration
OpenShiftEnterprise Kubernetes by Red Hat

๐Ÿ Docker Swarm โ€“ Container Orchestration with Ease


๐Ÿง  What is Docker Swarm?

Docker Swarm is Docker's native container orchestration tool that allows you to combine multiple Docker hosts into a single cluster (called a Swarm).

It provides high availability, load balancing, and fault tolerance โ€” all while keeping Docker's simplicity.


๐Ÿงฑ Why Use Swarm?

Without SwarmWith Swarm
Manual container deployment on each hostOne centralized command to deploy services
No built-in failoverAutomatic container recovery
No automatic load balancingBuilt-in service distribution
Difficult to scale across nodesEasy scaling with --replicas flag

๐Ÿ”ง Architecture of Docker Swarm

+----------------------------+
|       Swarm Manager       |
| (Leader & decision maker) |
+----------------------------+
             |
   -------------------------
   |           |           |
Worker 1   Worker 2   Worker 3  โ† Executes tasks/containers

๐ŸŽฏ Roles:

  • Manager Node:

    • Initializes and controls the swarm

    • Accepts service creation commands

    • Distributes tasks

  • Worker Nodes:

    • Execute containers (tasks)

    • Report status to manager


๐Ÿš€ Step-by-Step: Setting Up Docker Swarm

๐Ÿ–ฅ๏ธ Prerequisites:

  • At least 2 or more hosts (VMs or machines) with Docker installed

โœ… Step 1: Initialize the Swarm (Manager Node)

docker swarm init

๐Ÿ“ Output includes a docker swarm join command with a token for worker nodes.


โœ… Step 2: Join Worker Nodes to the Swarm

Run the command from manager output on each worker node:

docker swarm join \
--token <worker-token> \
<manager-ip>:2377

โœ… Once joined, these nodes become Swarm Nodes.


๐Ÿ“ฆ Deploying Applications with Docker Services

Instead of manually running containers on each host, you can now use Docker Services, which Swarm distributes automatically.


๐Ÿ› ๏ธ Traditional Method (not ideal):

docker run my-web-server
  • Must be run manually on each node

  • No auto-scaling, recovery, or balancing


docker service create --replicas=3 my-web-server
  • Run from manager node only

  • Creates 3 replicas of your web server

  • Distributes replicas across worker nodes

  • Monitors health & restarts if one fails


๐Ÿ“ก Swarm Service In Action

FeatureBenefit
--replicas=3Define how many instances to run
Manager node decidesAutomatically schedules containers to worker nodes
Auto-healingFailed containers restart automatically
Load balancingSwarm routes external traffic across all replicas

๐Ÿ” Swarm Management Commands

PurposeCommand
View swarm nodesdocker node ls
View running servicesdocker service ls
Inspect a servicedocker service inspect <service>
Scale a servicedocker service scale <svc>=5
Remove a servicedocker service rm <service>
Leave the swarm (worker)docker swarm leave
Leave the swarm (manager โ€“ force)docker swarm leave --force

โ˜ธ๏ธ Kubernetes โ€“ The King of Container Orchestration


๐Ÿงฑ What is Kubernetes?

Kubernetes (a.k.a. K8s) is an open-source container orchestration platform that automates:

  • ๐Ÿ”„ Deployment

  • โš–๏ธ Scaling

  • โ™ป๏ธ Self-healing

  • ๐Ÿ”„ Rolling Updates

Think of it as the "brain" behind managing containers at scale in production environments.


๐Ÿณ Docker vs โ˜ธ๏ธ Kubernetes

Docker CLIKubernetes CLI (kubectl)
docker run my-web-serverkubectl run my-web-server --replicas=1000
Manual scalingkubectl scale --replicas=2000 my-web-server
Manual updatekubectl rolling-update
Manual rollbackkubectl rolling-update --rollback
No auto-scalingBuilt-in auto-scaling capabilities

๐Ÿš€ What Can Kubernetes Do?

  • โœ… Deploy thousands of app instances with one command

  • ๐Ÿ“ˆ Automatically scale up/down based on load

  • ๐Ÿ” Rolling upgrades to update versions without downtime

  • โ†ฉ๏ธ Rollback to previous versions if something goes wrong

  • ๐Ÿงช Perform A/B testing (canary deployments)

  • ๐Ÿง  Monitor app health and restart failed containers

  • ๐ŸŒ Spread containers across multiple machines


๐Ÿ”— Docker & Kubernetes โ€“ What's the Relationship?

Kubernetes runs containers, and Docker was the original container runtime used.
But Kubernetes now supports multiple runtimes like:

  • ๐Ÿ”น Docker

  • ๐Ÿ”น containerd

  • ๐Ÿ”น CRI-O

  • ๐Ÿ”น rkt (deprecated)

๐Ÿ‘‰ Kubernetes manages containers; Docker runs them.


๐Ÿ—๏ธ Kubernetes Architecture

โš™๏ธ Cluster = Master + Worker Nodes

               +----------------------------+
               |       Master Node          |
               |  (Control Plane Components)|
               +----------------------------+
                    /         |         \
                   /          |          \
     +------------+   +------------+   +------------+
     | Worker Node|   | Worker Node|   | Worker Node|
     +------------+   +------------+   +------------+
         |                  |                |
     [Docker]          [containerd]      [CRI-O]
         |                  |                |
     [Pods/Containers] [Pods/Containers] [Pods/Containers]

๐Ÿง  Master Node Components (Control Plane)

ComponentDescription
API ServerExposes the Kubernetes API; all commands from kubectl go through this
etcdDistributed key-value store to store all cluster state
Controller ManagerWatches for changes (e.g., failed pods) and takes action
SchedulerDecides which node runs a new pod (based on resources, affinity, etc.)
Cloud Controller ManagerIntegrates Kubernetes with cloud services (optional)

๐Ÿ”ง Worker Node Components

ComponentDescription
kubeletAgent that runs on each node; takes instructions from the API server
Container RuntimeRuns the actual containers (e.g., Docker, containerd, CRI-O)
kube-proxyMaintains network rules and service discovery inside the cluster

๐Ÿ› ๏ธ kubectl โ€“ The Kubernetes Command-Line Tool

You interact with Kubernetes using kubectl (Kube Control):

๐Ÿ”ง Examples:

kubectl run my-web-server --image=web:v1 --replicas=1000
kubectl scale deployment my-web-server --replicas=2000
kubectl rolling-update my-web-server --image=web:v2
kubectl rolling-update my-web-server --rollback
kubectl get pods
kubectl describe deployment my-web-server

๐Ÿง  It communicates with the API Server, which updates the cluster state via etcd and schedules changes through the scheduler.


๐Ÿ“ฆ Kubernetes Objects Overview

ObjectPurpose
PodSmallest unit in Kubernetes (runs 1+ containers)
DeploymentManages replica sets, rolling updates
ServiceExposes pods as a single endpoint (with load balancing)
ReplicaSetEnsures desired number of pod replicas
NamespaceLogical partition within the cluster
ConfigMap & SecretStore config and sensitive data

โš–๏ธ Kubernetes = Production-Grade Orchestration

FeatureAvailable in Kubernetes? โœ…
Auto-scalingโœ… Horizontal Pod Autoscaler
Load balancingโœ… Internal and External
Rolling updatesโœ… Built-in
Rollbacksโœ… Instant
Health checksโœ… Liveness & Readiness Probes
Self-healingโœ… Pod restarts & replacement
Resource managementโœ… CPU & memory limits
0
Subscribe to my newsletter

Read articles from Arindam Baidya directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Written by

Arindam Baidya
Arindam Baidya