“The greatest tides of economy are powered by shell scripts in bland rooms.“

In the personal context, I generally use shell for most of my menial tasks. Acknowledgement is not enough though. I got to appreciate the significance of scripting firsthand at my job.

The entire cloud infrastructure of my team, spanning 60-70 microservices, is written as modularised scripts in a remote repository, with service-specific configurations appended in the scope of the service repositories. The entirety of code security and semantics analysis, code coverage, and code deployment, fault tolerance and diagnostics pipelines are automated across code shipping phases and triggered on code push, tag creation or manually if required.

The team is big. There are many people working on overlapping projects. Most basically, there has to be a written record of code and infra manipulation, right..? Writing infra pipelines allows for versioning them when they are hosted on VCS platforms, such as Github, Gitlab, Bitbucket, etc.

Opting for versioned and hosted scripts, over UI, to ensure standardisation and deterministic behaviour.

The key highlight of this, which I really love, is that no change, no matter how small or big, is unaccounted for. This way, there is no micromanagement and yet, there are managed records of version history.

Scripting covers all the cases, including but not limited to code being automatically moving on to the next shipping phase triggered from the successful deployment in the last one. (if all metadata is as expected; there is no undefined behaviour)

“Its not legacy tech if it is evergreen.“

Let me elaborate

Definitions

You give some instructions to the shell, the shell tells the OS to execute them.

You write a command in a shell window and run it. Stuff happens, computations are made and some output is written to the standard output stream. This pipeline of events though, contains multitude of intricacies.

The command you write may be an executable, i.e., written as a program, which may interact with the arguments provided, some files on the system, and with the OS kernel in the form of syscalls.
It was then added to the shell path to make it available to you as a command. You can write scripts which may contain multiple such commands, implemented as complex and branching logic, which may, in turn, be used in other scripts, in an attempt at modularity, better maintainability, etc.

This is a typical shell script.

#!/bin/bash
# Simple line count example, using bash
#
# Bash tutorial: http://linuxconfig.org/Bash_scripting_Tutorial#8-2-read-file-into-bash-array
# My scripting link: http://www.macs.hw.ac.uk/~hwloidl/docs/index.html#scripting
#
# Usage: ./line_count.sh file
# -----------------------------------------------------------------------------

# Link filedescriptor 10 with stdin
exec 10<&0
# stdin replaced with a file supplied as a first argument
exec < $1
# remember the name of the input file
in=$1

# init
file="current_line.txt"
let count=0

# this while loop iterates over all lines of the file
while read LINE
do
    # increase line counter 
    ((count++))
    # write current line to a tmp file with name $file (not needed for counting)
    echo $LINE > $file
    # this checks the return code of echo (not needed for writing; just for demo)
    if [ $? -ne 0 ] 
     then echo "Error in writing to file ${file}; check its permissions!"
    fi
done

echo "Number of lines: $count"
echo "The last line of the file is: `cat ${file}`"

# Note: You can achieve the same by just using the tool wc like this
echo "Expected number of lines: `wc -l $in`"

# restore stdin from filedescriptor 10
# and close filedescriptor 10
exec 0<&10 10<&-

The she-bang, here, #!/bin/bash, It tells the system which interpreter to use to run this file. It's the script's own instruction manual.

Then, there is the concept of permissions. Categorised into read(r), write(w) and execute(x), the current user must have the necessary set of permissions to perform desired action. for eg, a user may have permission to read and execute a certain script, but not to modify it.

~ l | grep aatir.sh
> -r--rw-r--  1 aatir aatir    0 Jun 22 20:46 aatir.sh

~ chmod u+w ./aatir.sh 

~ l | grep aatir.sh 
> -rw-rw-r--  1 aatir aatir    0 Jun 22 20:46 aatir.sh

Making the script available in the shell globally, will require adding it to the path of the shell, i.e., $PATH variable for the shell, (an analogy can be a phone directory), such that the shell may know, where to look for the script, when required.

export PATH="$GOPATH:$IDEA_PATH:$POETRY_PATH:$PATH" # other paths may be concatenated here

1. Move the script: Move hello.sh to a directory already in the $PATH (like /usr/local/bin*)*.

2. Modify the path: Add a custom scripts directory (e.g., ~/bin) to your $PATH in your .bashrc or .zshrc file. This is the preferred method for personal scripts.

Execution

The Core Loop (REPL): Read -> Evaluate -> Print -> Loop. The shell is in an infinite loop waiting for your input.

There are certain commands which run inside shell’s own process and others which are written as external programs and are loaded from the disk.

• Built-ins (cd, echo, export): Part of the shell itself. They run instantly inside the shell's own process. No new program is started. Just as you would expect, these are basic commands, which do not require external or constant updates.

• External Commands (ls, grep, curl): Separate programs on your disk. The shell has to find them and launch them.

The Process Lifecycle: fork() and exec()

• fork() - The Clone: When you run ls, the shell doesn't become ls. It first creates an almost identical copy of itself—a child process. This is fork().

• exec() - The Brain Swap: The new child process then completely replaces itself with the ls program. Its memory, its code—everything is now ls. This is exec().

• wait() - The Patient Parent: The original parent shell waits for the child process (ls) to finish. Once done, the parent shell prints the prompt again, ready for the next command.

Connecting to the Kernel: The Role of System Calls (Syscalls)

• Explain that user programs (like ls or the shell) can't directly access hardware (like the hard drive or network card). It would be chaos.

A Syscall is an API of the OS kernel.

The kernel is the gatekeeper.
A syscall is the official, secure way for a program to request a service from the kernel.

When a program needs something done that requires privileged access, it doesn't do it. It asks the kernel to do it on its behalf. It fills out a "request form" (the syscall with its arguments) and hands it to a special CPU instruction.
The kernel then takes over, validates the request, performs the action in its own secure environment, and hands back the result.

Example Walk-through: When ls runs, it doesn't read the disk itself. It uses syscalls like:

• open(): "Kernel, please open this directory for me."

• readdir(): "Kernel, please read the contents of this directory."

• write(): "Kernel, please write this text to the screen."

• close(): "Kernel, I'm done with this directory."

Varieties of shell

If you are wish to specify some difference among the shells based on their interaction with the OS kernel, or the set of syscalls, you would not find any. Syscalls provide a standardized way of interacting with the kernel and the same set of functions have to be used by every shell implementation.
The differences are almost entirely at the application layer—in the features, syntax, and philosophy they are built with.

• The Patriarch: sh (Bourne Shell)

  • The original. POSIX-compliant (POSIX compliance, in the context of operating systems and software, means adhering to the standards defined by the Portable Operating System Interface (POSIX) specifications; the baseline for all portable scripting). When you write #!/bin/sh, you're aiming for maximum compatibility.

• The Workhorse: bash (Bourne-Again Shell)

  • The long-time default on most Linux systems.

  • What it adds: Command history, tab completion, arrays, brace expansion ({1..10}), etc. The "de facto" standard for a long time.

• The Modern Contender: zsh (Z Shell)

  • The standard contender to bash.

  • Key Features: Powerful auto-completion, advanced globbing (**/*), and a massive plugin ecosystem (Oh My Zsh, Prezto) that makes it highly customizable.

• The User-Friendly Futurist: fish (Friendly Interactive Shell)

  • Its philosophy: "Smart and user-friendly, out of the box."

  • Key Features: Autosuggestions (based on history), syntax highlighting by default, no complex configuration needed.

  • Crucial Caveat: Its scripting language is not POSIX-compliant. A bash script may not run correctly in fish. This is a trade-off between interactivity and portability*.*

Then again, delving into the vertical, some shells, are better suited for certain categories of tasks.

Learning about these different shells, I wondered whether there would be any minute internal differences which fares one kind of shell better than another for a given set of tasks. And, I was right.
While shells might look similar when running basic commands, their internal architecture leads to significant performance differences, especially in scripts that run thousands of operations. These differences come down to one thing: how cleverly they avoid creating new processes.

While they all use the same set of syscalls, they can be differentiated by the manner, frequency, and arrangement in which they make those calls. This directly impacts performance metrics like CPU usage and process traffic.

“The fastest syscall is the one you don't make.“

$ ./shellbench -s sh,bash,ksh,mksh,posh,zsh sample/count.sh sample/output.sh
------------------------------------------------------------------------------------------------
name                                   sh       bash        ksh       mksh       posh        zsh
------------------------------------------------------------------------------------------------
count.sh: posix                 1,034,369    248,929    282,537    364,627    411,116    577,090
count.sh: typeset -i                error    237,421    288,133    341,660      error    593,124
count.sh: increment                 error    272,415    443,765    350,265      error    835,077
output.sh: echo                   279,335    121,104    375,175    179,903    201,718     59,138
output.sh: printf                 277,989    118,461    209,123        180        179     63,644
output.sh: print                    error      error    281,775    182,388      error     63,006
------------------------------------------------------------------------------------------------
* count: number of executions per second

Refer to this application docs for further details: Shellbench

By choosing to implement a feature as a built-in*, or by providing a clever **arrangement of syscalls to simulate a feature like process substitution, shell designers make critical trade-offs between speed, features, and POSIX compliance. For a system administrator writing a boot script*, minimizing process traffic with dash is key. For a developer writing a complex *data-munging script**, bash or zsh's ability to avoid temporary files might be the winning factor.*

Advantages of shell scripting

Pillar 1: Automation & Repeatability:

Any task you do once, you might need to do again. The CLI allows you to script these tasks, ensuring they are done exactly the same way every time. This is the foundation of DevOps and Infrastructure as Code (IaC).

Suppose you, along with your team, are working on a service or an application and you want to push it as an image to an AWS ECR repository.
Keep in mind that you should tag your image, for versioning purposes.

There are several things to keep in mind.

In your current terminal session, you should be logged in to your AWS account; your container management system (eg. docker) must be logged into the ECR registry; and, the correct ECR repository must be present.
Then, you tag your local image with the repo url and a new tag and push it to the ecr.
It is also possible and indeed, recommended, that the configurations for your images be standardized and shared across the team, and that they may also be versioned.

A generic example for the purpose of this article:

#!/bin/bash

# --- Configuration ---
# In a real project, you'd load these from a file, but we define them here for simplicity.
AWS_ACCOUNT_ID="123456789012"
AWS_REGION="us-east-1"
ECR_REPOSITORY_NAME="my-awesome-app"

# --- Script Logic ---

# Stop the script if any command fails. This is our safety net.
set -e

# Check if we got the two required arguments: the local image and the new tag.
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <local-image-name> <new-version-tag>"
    echo "Example: $0 my-app:latest v1.2.0"
    exit 1
fi

LOCAL_IMAGE=$1
NEW_TAG=$2

# Construct the full ECR image URI.
ECR_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
FULL_IMAGE_NAME="${ECR_URI}/${ECR_REPOSITORY_NAME}:${NEW_TAG}"

echo "[INFO] Starting ECR push for image: $FULL_IMAGE_NAME"

# Step 1: Log Docker into the Amazon ECR registry.
# This command securely retrieves a temporary access token from AWS and pipes it
# to the 'docker login' command. The password is never shown on screen.
echo "[STEP 1/3] Authenticating Docker with AWS ECR..."
aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "$ECR_URI"

# Step 2: Tag the local image with the ECR repository URI.
# Docker needs this specific tag format to know where to push the image.
echo "[STEP 2/3] Tagging local image..."
docker tag "$LOCAL_IMAGE" "$FULL_IMAGE_NAME"

# Step 3: Push the newly tagged image to ECR.
# This is the step that uploads your image to the cloud.
echo "[STEP 3/3] Pushing image to ECR..."
docker push "$FULL_IMAGE_NAME"

echo ""
echo "--------------------------------------------------"
echo "✅ Successfully pushed image to ECR:"
echo "$FULL_IMAGE_NAME"
echo "--------------------------------------------------"

Another example,
Check if a web server (Nginx) is running. If not, restart it and send a notification.

#!/bin/bash

# Check if the nginx service is active
if ! systemctl is-active --quiet nginx; then
    echo "Nginx is down! Attempting to restart..."

    # Try to restart the service
    sudo systemctl restart nginx

    # Verify if it came back up
    if systemctl is-active --quiet nginx; then
        echo "Nginx restarted successfully."
        # Optional: Add a command here to send a Slack/Discord notification
    else
        echo "CRITICAL: Nginx failed to restart!"
    fi
else
    echo "Nginx is running smoothly."
fi

Pillar 2: Composability (The Unix Philosophy):

"Write programs that do one thing and do it well."

This is the primary reason I appreciate the power of scripting. You can achieve so much just by stringing together basic commands with good judgement.

• Constructing a script to achieve a complex logic is not rocket science. It is modularising the logic into components or units and arranging these units in a manner which achieves the end result. This arrangement provides abstraction, maintainability and better diagnostics and readability.

• Lets understand the significance of this pattern with an example.

  You want to delete all the temporary files from a certain directory, which are at least 15 days old.

    # We'll create old and new temp files, plus other files that should NOT be deleted.
    ➜  scripting touch -d "20 days ago" old_log_file.tmp
    ➜  scripting touch -d "30 days ago" session_cache.tmp
    ➜  scripting touch -d "2 days ago" new_report.tmp
    ➜  scripting touch ./new_log_file.tmp 
    ➜  scripting touch "important_config.conf"
    # view the modification dates, which is key.
    ➜  scripting l
    total 12K
    drwxrwxr-x 2 aatir aatir 4.0K Jun 23 18:14 .
    drwxrwxr-x 5 aatir aatir 4.0K Jun 22 23:37 ..
    -rw-rw-r-- 1 aatir aatir    0 Jun 23 18:12 important_config.conf
    -rw-rw-r-- 1 aatir aatir    0 Jun  7 18:11 less_older_file.tmp
    -rw-rw-r-- 1 aatir aatir    0 Jun 23 18:14 new_log_file.tmp
    -rw-rw-r-- 1 aatir aatir    0 May 24 18:11 older_file.tmp
    -rw-rw-r-- 1 aatir aatir    0 Jun  3 18:11 old_file.tmp
    -rw-rw-r-- 1 aatir aatir    3 Jun 22 23:48 testing.ps1
    # let's run `find` by itself first to see what it finds.
    ➜  scripting find ./ -name "*.tmp"  
    ./new_log_file.tmp # as you can see, it lists all the files
    ./less_older_file.tmp
    ./old_file.tmp
    ./older_file.tmp
    # listing all the files with the given pattern which are at least 15 days old
    ➜  scripting find ./ -name "*.tmp" -mtime +14
    ./less_older_file.tmp
    ./old_file.tmp
    ./older_file.tmp
    # provide this list to the `xargs` command, which outputs it to the `rm` command
    ➜  scripting  find ./ -name "*.tmp" -mtime +14 | xargs rm         
    # all the temporary files from at least 15 days ago, have been removed
    ➜  scripting l
    total 12K
    drwxrwxr-x 2 aatir aatir 4.0K Jun 23 18:16 .
    drwxrwxr-x 5 aatir aatir 4.0K Jun 22 23:37 ..
    -rw-rw-r-- 1 aatir aatir    0 Jun 23 18:12 important_config.conf
    -rw-rw-r-- 1 aatir aatir    0 Jun 23 18:14 new_log_file.tmp
    -rw-rw-r-- 1 aatir aatir    3 Jun 22 23:48 testing.ps1
  

  Deconstructing the Command: find . -name "*.tmp" -mtime +14 | xargs rm

  While this looks complex, fundamentally, it is pretty straightforward.

  1. find : This is the star of the show. It's designed for exactly this kind of search. Let's break down its arguments:

     • ./: Search in the current directory.

     • -name "*.tmp": This is a test. It looks for items whose name matches the pattern *.tmp.

     • -mtime +14: This is the time-traveling part. It's a second test that looks for files whose modification time is more than (+) 14 full 24-hour periods ago (i.e., 15 days or older).

The find command only outputs the names of files that pass all of its tests.

2. | : The pipe takes the list of matching files from find and sends it to the next stage.

3. xargs (The Argument Formatter): It takes the newline-separated list from find and transforms it into a space-separated list for rm.

4. rm : The rm command receives the formatted list and dutifully removes the files it was given.

Now that the units are explained, this seems dead simple right..? Now, imagine there are thousands or millions of tmp files and more are added on a weekly basis, and this is common for hundreds of nodes you have setup for a building-wide architecture, lets say.
The second last compound statement executed, is a single line, yet it can deal with those millions of temporary files without any manual intervention.
You can put this statement in a daemon or a cron job and automate the entire thing.

This, right here, is the power of scripting.

Pillar 3: Universal Access:

You can't right-click on a remote server in a data center or an embedded IoT device. The shell, accessed via SSH, is the universal language for managing remote systems. Mention that even on systems without a GUI (like most servers), the CLI is always there.

➜ aatir ssh -p 2222 -i ~/.ssh/my-aws-key.pem ec2-user@18.221.123.456 "df -h /"
# -p is for the port
# i provide my pem file as a means for authentication
# i have the address of the remote server
# lastly, the command i want to execute once a secure session is established
# writing the command in this manner, immediately breaks the session after the command 
# execution
➜ aatir rsync -avz ./build/ user@your_server:/var/www/my-app && ssh user@your_server "sudo systemctl restart my-app.service"
# rsync -avz: A highly efficient tool for syncing files. It's faster and more robust than scp.# 
# &&: A logical AND. The second command only runs if the first one succeeds.
# ssh user@... "...": Executes a command on the remote server without needing to log in interactively

Pillar 4: The GUI is an Abstraction:

Use your examples here. When you use Docker Desktop or a Git GUI (like Sourcetree), they are often just executing docker run ... or git commit ... commands under the hood. The CLI gives you direct access to the full power of these tools, without the limitations of the GUI.

Everything from software development and deployment, networking (management, provisioning, diagnostics and security), system administrations to infrastructure provisioning, secure file transfer, etl pipelines, etc., can be set up with compound scripts, which build upon basic logic and make use of existing scripts and executables.

It is all about placing one brick on top of the other. This is just implementing isolated logic built upon abstraction. Again, as I have mentioned before (in earlier articles), the science of robust software development is building your logic around the end user requirements. Have a clear picture of your data flow and build your scripts around it (obviously, if required).

Beyond Linux terminal

The philosophy of promoting CLI, isn't limited to Linux.
Windows PowerShell is an incredibly powerful object-oriented shell that is the standard for automating Windows Server and Azure.

Just like UNIX based commands, such as ls, cat, echo, cd, Microsoft has corresponding commands such as Get-ChildItem, Get-Content, Write-Output, Set-Location, etc., respectively.

Check out this article for further details: Powershell Commands Table

Lets consider an example,
Monitor the free space on the C: drive and If it falls below a 15% threshold, automatically clean out the user's temporary files older than 7 days to prevent system issues, and log the action.

#Requires -RunAsAdministrator

<#
.SYNOPSIS
    Monitors disk space and cleans up old temp files if space is low.
.DESCRIPTION
    This script checks the percentage of free space on the C: drive.
    If it is below the defined threshold, it deletes files in the user's
    temp directory that haven't been modified in the last 7 days.
#>

# --- Configuration ---
$driveLetter = "C"
$thresholdPercentage = 15
$cleanupPath = "$env:USERPROFILE\AppData\Local\Temp"
$cleanupDays = 7

# --- Logic ---
Write-Host "Starting disk space check for drive $driveLetter..." -ForegroundColor Cyan

# Get volume information as an object
$volume = Get-Volume -DriveLetter $driveLetter

# Calculate the percentage of free space
$percentFree = [math]::Round(($volume.SizeRemaining / $volume.Size) * 100, 2)

# Check if the free space is below the threshold
if ($percentFree -lt $thresholdPercentage) {
    Write-Host "WARNING: Drive $driveLetter is low on space ($percentFree% free)." -ForegroundColor Yellow
    Write-Host "Threshold is $thresholdPercentage%. Starting cleanup of files older than $cleanupDays days in '$cleanupPath'."

    # Get files older than the specified number of days
    $oldFiles = Get-ChildItem -Path $cleanupPath -Recurse -Force | Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-$cleanupDays) }

    if ($oldFiles) {
        # Clean up the files and log the action
        $totalSize = ($oldFiles | Measure-Object -Property Length -Sum).Sum
        $sizeInMB = [math]::Round($totalSize / 1MB, 2)

        Write-Host "Found $($oldFiles.Count) old files, totaling $sizeInMB MB. Deleting..." -ForegroundColor Yellow

        # The -WhatIf switch simulates the deletion. Remove it to perform the actual deletion.
        $oldFiles | Remove-Item -Force -Recurse -WhatIf

        Write-Host "Cleanup simulation complete. Re-run without '-WhatIf' on the Remove-Item command to perform the action." -ForegroundColor Green
        # Add-Content -Path "C:\Logs\cleanup.log" -Value "($(Get-Date)) - Low disk space triggered. Cleaned up $sizeInMB MB of temp files."
    } else {
        Write-Host "Low disk space detected, but no old files found to clean up in '$cleanupPath'." -ForegroundColor Red
    }

} else {
    Write-Host "OK: Drive $driveLetter space is healthy ($percentFree% free)." -ForegroundColor Green
}

Terminal UIs

Terminal user interfaces (TUIs), also known as text-based user interfaces, are a type of user interface that utilise text characters, often with the aid of Unicode characters for visual elements, to interact with a computer system. They are distinct from graphical user interfaces (GUIs) which rely on visual elements like windows, icons, and a mouse. TUIs are typically accessed through a terminal emulator or directly on text-based hardware.

The Middle Ground, Text-based User Interfaces like htop (a system monitor) or lazygit (a git TUI), ranger, bashtop, ncmcpp, etc. are very popular and widely adopted.
They offer a visual experience inside the terminal, blending the best of both worlds.

Beyond the Shell – When to Reach for a Bigger Toolbox

The Tipping Point: Acknowledge that shell scripting has limits. When should you stop?

• Complex data structures (JSON, nested objects).

• Heavy-duty error handling and logic.

• Making API calls.

• Needing a user-friendly CLI with flags and sub-commands.

Better Tools for the Job:

• Python: The king of "glue code." Perfect for scripts that need to parse JSON, talk to APIs (requests), or build robust CLIs (argparse, Typer, Click).

• Go / Rust: The choice for high-performance, compiled, single-binary CLI tools. Great for system utilities where speed and type safety matter.

• Modern Terminal Enhancements: Tools that aren't shells but improve the experience: fzf (fuzzy finder), ripgrep (fast grep), bat (a better cat), exa (a better ls).

Conclusion

To sum it up, we have witnessed that the command line is the engine driving modern technology, complementary to the cute UI you would want to entice your user with.

• Automation: The ability to script any task, from system health checks to full application deployments, ensuring it runs perfectly every time and eliminating costly human error.

• Composability: The elegant Unix philosophy of chaining small, single-purpose tools together with pipes (|) to solve incredibly complex problems with a single line of code.

• Universal Access: The power to securely control any machine, anywhere in the world, from a server in a data center to an embedded device, often without a graphical interface in sight.

When work needs to be serious, scalable, and repeatable, the Command Line Interface becomes essential. It is the right tool for the job. The reliability of your final product—be it a web application, a mobile app, or a cloud service—is built upon a foundation of robust backend management, security audits, and deployment pipelines. These are the domains where the CLI reigns supreme, acting as the professional's cornerstone for building quality from the ground up.

The might of the shell is not reserved for an elite few. It's available to you, right now, in your terminal. Try it.