Learning Python Automation and Docker Containers

Introduction

Hello there! I was unable to write any article for the past 2 weeks due to a few reasons, but the learning process didn't stop. For the past two weeks, I have been revising and exploring the capabilities of Python and understanding Docker in a more comprehensive way, which is one of the most important tools in the modern software development era. So in this article, I will be discussing the experiments I have been doing with Python and my experience learning Docker.

Python Revision and Scripting

Python is widely used in DevOps due to its versatility. It is used for automation, CI/CD pipelines, infrastructure as code (IaC), and more. One of the most common applications of Python is scripting, and the extensive library support makes Python an easy choice for engineers when it comes to automation.

In my first semester, Python was part of our curriculum, but it wasn't taught extensively. So first, I revised the basic syntax and fundamental concepts of Python such as lists, sets, dictionaries, functions, etc., and practiced them using simple programs.

After revising the basic syntax and concepts, it was time to learn important stuff like modules, automation, and scripting. The libraries I learned about that are really important and useful in scripting are os, subprocess, and argparse. These modules are essential for automating Linux system tasks. I learned different functions of these modules such as os.system(), os.mkdir(), subprocess.run([]), etc. To implement these functions in a real-world script, I created an Ubuntu VM and wrote a user management script to perform addition and removal of users and groups.

#!/usr/bin/python3

import argparse
import subprocess
import logging
import getpass

logging.basicConfig(
    filename="logs/actions.log",
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

# Main command runner
def run_cmd(cmd):
    try:
        result = subprocess.run(cmd, check=True, text=True, capture_output=True)
        logging.info(f"Command succeeded: {' '.join(cmd)}")
        print(result.stdout.strip())
    except subprocess.CalledProcessError as e:
        logging.error(f"Command failed: {' '.join(cmd)}\n{e.stderr}")
        print(f"Error: {e.stderr.strip()}")

# Argument parser
parser = argparse.ArgumentParser(description="Linux User & Group Management")

parser.add_argument('--add-user', help="Add a new user")
parser.add_argument('--group', help="Assign user to group (with --add-user)")
parser.add_argument('--del-user', help="Delete a user")
parser.add_argument('--add-group', help="Create a new group")
parser.add_argument('--del-group', help="Delete a group")
parser.add_argument('--list-users', action='store_true', help="List all system users")
parser.add_argument('--change-pass', help="Change password of a user")

args = parser.parse_args()

# Actions
if args.add_user:
    cmd = ["sudo", "useradd", args.add_user]
    if args.group:
        cmd.extend(["-G", args.group])
    run_cmd(cmd)

if args.del_user:
    run_cmd(["sudo", "userdel", "-r", args.del_user])

if args.add_group:
    run_cmd(["sudo", "groupadd", args.add_group])

if args.del_group:
    run_cmd(["sudo", "groupdel", args.del_group])

if args.list_users:
    run_cmd(["cut", "-d:", "-f1", "/etc/passwd"])

if args.change_pass:
    password = getpass.getpass("Enter new password: ")
    confirm = getpass.getpass("Confirm new password: ")
    if password == confirm:
        proc = subprocess.Popen(["sudo", "passwd", args.change_pass], stdin=subprocess.PIPE)
        proc.communicate(input=f"{password}\n{password}\n".encode())
        logging.info(f"Password changed for {args.change_pass}")
    else:
        print("Passwords do not match.")
        logging.warning(f"Password mismatch for user {args.change_pass}")

# Help command if no arguments are passed
if len(vars(args)) == 0:
    parser.print_help()

With the help of ChatGPT, I enhanced the script using the getpass module for the change password action, and it was working as expected.

Python Automation

One of the main use cases of Python in DevOps is automation. I came to know about two libraries that are primarily used for automation: Boto3 and Fabric. Boto3 is the official SDK that is used to interact with AWS services and simplifies the process of creating, deleting, and configuring different AWS services. While learning how to use Boto3, it became very clear to me that to make use of boto3 or even any automation tool, we first need to fully understand the manual process that we are trying to automate. So the knowledge of creating EC2 instances, uploading files to S3 buckets, and creating RDS instances using the AWS console with all the steps was crucial in order to write automation scripts using Boto3.

Using the Boto3 documentation, I created a few scripts to create EC2 instances, upload files to S3, and perform health checks on running EC2 instances. Understanding the responses of each call like ec2.describe_instances() was tricky at first as there was so much information to first understand and then filter out for what you actually want, such as instance ID and status. Formatting the responses using the pprint library helped in this process.

Docker Deep Dive

Earlier, to learn and work with different technologies like Jenkins and AWS, I learned about Docker but not in depth. But for DevOps, saying Docker is an essential tool would be an understatement, so learning Docker extensively is a must. So first, I checked what the main topics to learn Docker were, and Claude provided me with a list:

Docker Overview (bigger picture)
Docker Architecture
Images
Containers
Dockerfile
Docker Networking
Volumes and Storage
Docker Compose
Docker Registry

Docker solves the "But it works on my machine" problem that was repeated millions of times in the past. The main reason for this problem is inconsistencies in development environments such as differences in operating systems, application dependency issues, etc. To overcome this issue, containerization is used—a lightweight form of virtualization that packages an application and all its dependencies into a portable, executable unit called a container.

Docker operates on a client-server architecture with three primary components:

Docker Engine
Docker Client
Docker Registry

I learned in-depth about important terminologies like Dockerfile, Images, and Containers. I also learned about the flow of Docker: Dockerfile is used to build read-only templates which are called Images that contain everything needed to run the application. These images are used to create Containers in which applications run.

I practiced Docker on an EC2 instance by cloning a few simple projects and trying to containerize them. I wrote Dockerfiles for different projects such as Java applications and Python applications—getting base images, creating working directories, copying source code, and installing dependencies. But the main challenge was when I tried to create a Dockerfile for my own Node project. When I was able to build its image successfully and deploy its container and could access my project, I was delighted.

I proceeded to learn more important concepts such as networking in Docker, how Docker containers communicate with each other and with external networks using network drivers such as Bridge, Host, Overlay, and None network. After deploying a container for any project, when I was trying to restart the container, the data was being lost from the application, and then I came to know about the importance of Volumes and Storage. Volumes and Storage solve the challenge of data persistence in containerized applications. By creating and mounting the containers to named volumes, even if we stop, remove, or restart the container, the data will be stored in the attached volume.

While learning these topics, I noticed one thing: this is a lot of manual work for such simple projects—pulling images, creating Dockerfiles, building application images, running containers, all actions through manual commands. While commands are great for understanding the fundamental work, for real work at companies, this would be too time-consuming to configure hundreds of containers. The solution to this problem is Docker Compose. Docker Compose is a tool for defining and running multi-container Docker applications using YAML configuration files. I learned about the syntax and different components of the docker-compose.yml file such as services, networks, and volumes. To make use of it, I first removed all the containers from the previous projects and then configured a docker-compose.yml file to build a two-tier Flask application with a Flask backend and MySQL database.

At last, I understood Docker Registry—a centralized repository for storing, managing, and distributing Docker images. After logging into DockerHub using docker login, I learned to tag images and to push and pull from DockerHub. I pushed my two-tier-flask-app image to DockerHub, which I can use in the future to practice more.

Resources I Used

Challenges I Faced

1️⃣ Python interpreter not working in Git Bash

The Python interpreter wasn't working properly in Git Bash when trying to run my automation scripts due to path recognition and environment variable issues.

Solution: Switched to PowerShell which resolved the issue completely.

2️⃣ Wrong path error after deploying Docker container for canvas-assignment application

Encountered path errors that prevented the React application from loading correctly in the containerized environment.

Solution: Replaced "homepage" line in package.json to "." and forced the PUBLIC_URL environment variable in Dockerfile.

3️⃣ Difficulty in creating health check in docker-compose.yml file

Had trouble understanding the syntax and getting the health check commands right for multi-container application monitoring.

Solution: Referred to Docker Compose documentation and configured proper health checks that monitor both application status and database connectivity.

What's Next?

For the next week, I am planning to write some more automation scripts using Python to understand more about scripting. I am also planning to combine all the Docker knowledge and use all the components to create a comprehensive project and then revise all these concepts.

Let's Connect!

🔗 My LinkedIn 🔗 My GitHub

If you have any recommended resources, better approaches to my challenges, or insights, I'd love to hear them! Drop your thoughts in the comments.

Have a wonderful day!

Python Automation Scripts & Multi-Container Docker Applications

Table of contents