Advanced Shell Scripting Concepts

Shell scripting is a powerful way to automate daily activities on a Linux server. In this blog, we will explore the basics of shell scripting, different shell executables, permission management, useful commands, and advanced concepts crucial for infrastructure maintenance and configuration management.

What is Shell Scripting?

Shell scripting involves writing a series of commands in a script file to automate tasks. These scripts can perform a variety of functions, from simple file manipulations to complex system administration tasks.

Manual Pages

To understand how to use different commands and their options, the man (manual) command is invaluable:

man <command>

This provides detailed information about the command and its available flags.

Different Shell Executables

There are several shell executables available in Linux:

• sh (Bourne Shell): An early and simple shell.

• bash (Bourne Again Shell): The most widely used shell, with more features than sh.

• dash (Debian Almquist Shell): A POSIX-compliant shell, often used for system scripts due to its speed.

• ksh (Korn Shell): An older shell that is largely obsolete.

Bash vs. sh vs. dash

• bash is the most commonly used shell, known for its advanced features and user-friendliness.

• sh was traditionally linked to bash, but many modern systems, like Ubuntu, link sh to dash for faster script execution.

• dash is preferred for system scripts in some distributions due to its minimalism and speed.

• ksh is considered obsolete and is rarely used in modern systems.

Basic Shell Scripting Commands

Viewing and Executing Shell Files

• View a script:

    cat <filename>
  

• Execute a script:

    sh <filename> or ./<filename>
  

Permission Management

When you create a shell script, you need to grant execution permissions. Granting permissions in Linux is not straightforward, but using chmod simplifies the process.

• Change file permissions:

    chmod 777 <filename>
  

  Here, 777 grants read, write, and execute permissions to everyone. The digits represent permissions for the owner, group, and others, respectively:

  • The first digit is for the owner.

  • The second digit is for the group.

  • The third digit is for all users.

The permissions are represented as follows:

• Read (r): 4

• Write (w): 2

• Execute (x): 1

So, 777 means:

• Owner: 4 (read) + 2 (write) + 1 (execute) = 7

• Group: 4 (read) + 2 (write) + 1 (execute) = 7

• Others: 4 (read) + 2 (write) + 1 (execute) = 7

History and Directory Management

• View command history:

    history
  

• Create a directory:

    mkdir <directory_name>
  

Sample Shell Script

Here is an example of a simple shell script:

#!/bin/bash

# Create a directory
mkdir dir1

# Create files
touch s1 s2

Execute the script with:

sh script.sh

or

./script.sh

Comments in Shell Script

Use # to add comments in your shell scripts for better readability:

#!/bin/bash
# This is a comment
echo "Hello, World!"

Focus of Shell Scripting

• Infrastructure Maintenance

• Interacting with Git

• Configuration Management

Advanced Concepts in Shell Scripting

Metadata in Scripts

Include metadata in your scripts for better documentation:

# Author: Your Name
# Date: YYYY-MM-DD
# Description: Script description

Debugging and Process Management

• Enable debug mode:

    set -x
  

• List running processes:

    ps -ef
  

• Filter processes:

    ps -ef | grep <pattern>
  

Using Pipes

Pipes allow you to pass the output of one command as input to another. However, pipes only work with standard output (stdout). For example:

date | echo "date is:"

This command won't work as expected because echo does not accept input from the pipe; it only prints the string. The correct use of pipes would be:

date | awk '{print "Date is:", $0}'

This will prepend "Date is:" to the output of the date command.

Using awk and grep

• grep retrieves entire lines containing a pattern:

    grep <pattern> <file>
  

  awk retrieves specific columns:

    awk -F ' ' '{print $4}' <file>
  

Example:

ps -ef | awk -F ' ' '{print $2}'
grep name test | awk -F ' ' '{print $4}'

The difference between grep and awk is that grep finds lines matching a pattern, while awk can extract specific columns from the output.

Switching User

• Switch to root user:

    sudo su -
  

Shell Script Modes

• Debug mode:

    set -x
  

• Exit on error:

    set -e
  

• Exit on error with pipefail:

    set -o pipefail
  

• Combined modes:

    set -exo
  

Useful Network Commands

• Download files with curl:

    curl <url>
  

• Download files with wget:

    wget <url>
  

Difference between curl and wget

• curl is used to transfer data from or to a server, supporting multiple protocols (HTTP, FTP, IMAP, etc.). It is often used for interactions with APIs and can output the data directly to the terminal.

    curl http://example.com
  

• wget is used to download files from the web. It supports HTTP, HTTPS, and FTP protocols and is often used to download large files.

    wget http://example.com/file.tar.gz
  

Searching Files

• Search for files:

    find / -name <filename>
  

Loops and Traps

• Using traps:

    trap 'rm -rf *' SIGINT
  

  This command traps the SIGINT signal (usually generated by pressing Ctrl+C) and executes the specified command.

Conclusion

Shell scripting is an indispensable tool in the DevOps toolkit, enabling automation of routine tasks, efficient system administration, and seamless integration with various DevOps tools. By mastering the basics and advancing to more complex scripts, you can significantly enhance your productivity and system reliability.

Feel free to dive deeper into each topic, experiment with different commands, and tailor your scripts to meet your specific needs. Happy scripting!