Zero to Linux: An Intensive Crash Course

Introduction:

• The video aims to be a comprehensive guide to Linux for DevOps engineers, covering topics from basic to advanced in one shot.

• Initial topics to be covered include internet basics, Linux introduction, installation methods, and essential commands.

Internet Basics:

• The internet works through optical fibers or fiber cables laid underwater connecting data centers.

• Data centers are large locations housing numerous computers for data storage and transmission.

• Data is transferred through these cables owned by companies like AT&T and Jio.

• Users pay these companies for internet access, referred to as internet recharge or data pack.

• When you request data, it travels through these fiber cables from the data center to your device.

• Local internet service providers (ISPs) like Your Broadband or Airtel Xstream provide the final connection to your home.

Servers:

• A server is fundamentally a computer designed to serve information.

• Different types of servers are defined by the type of information they serve:

  • Email Server: Serves emails.

  • File Server: Stores files and provides access to them.

  • Database Server: Manages and allows interaction with databases.

  • Application Server: Runs applications, like websites (e.g., facebook.com, youtube.com).

  • Web Server: Serves static content such as images and HTML pages.

• A crucial distinction is between servers and clients:

  • Server: Serves information.

  • Client: Requests information from the server. Examples include phones and laptops with browsers.

• Process of accessing a website like:

  1. Enter URL: You type a web address (URL) into your browser's address bar.​

  2. DNS Lookup: The browser translates the domain name (e.g., www.example.com) into an IP address using DNS servers.​

  3. Establish Connection: A TCP connection is set up between your browser and the web server hosting the site.​

  4. Send HTTP Request: Your browser sends an HTTP request to the server for the desired web page.​

  5. Receive Response: The server sends back the requested web page data.​

  6. Render Page: Your browser processes and displays the web page for you to view and interact with.

Introduction to Linux:

• Linux is an operating system where applications can run, allowing file creation, browser installation, and coding.

• Linux is largely open source and free to use.

• Multiple ways to use Linux are available:

  • Dual booting alongside Windows.

  • Installing Linux within Windows using WSL (Windows Subsystem for Linux).

  • Using a virtual machine (e.g., VirtualBox).

  • Creating a virtual machine on a cloud platform (AWS, Azure, GCP).

  • Using Vagrant, a tool by HashiCorp for managing virtual environments.

• Common Linux distributions include Ubuntu and Kali Linux.

• Linux is used for coding and provides tools for remote server management like RDP (Remote Desktop Protocol) and SSH (Secure Shell).

Boot Loader:

• The boot process starts when the power button is pressed, energizing the CPU and hard disk.

• The operating system, including the kernel, is stored on the hard disk.

• The kernel contains processes, one of the first being the boot loader.

• The boot loader is a process within the kernel responsible for running the necessary files to start the operating system.

• GRUB (GNU GRand Unified Bootloader) is a common boot loader in Linux.

• The basic Linux system architecture involves the kernel at the center, the shell as an interface to the kernel, and utilities/applications interacting with the shell.

Processes in Linux:

• Linux processes have various states, including running, sleeping, terminated, and zombie.

• Linux Process States

  In Linux, processes can exist in various states, each representing a specific condition in their lifecycle:

  1. Running (R): A process is in this state when it is either currently executing on the CPU or is ready to execute.​

  2. Sleeping:

     • Interruptible Sleep (S): The process is waiting for an event or condition to proceed but can be interrupted by signals.​ Eg. When a process waits for user input.

     • Uninterruptible Sleep (D): The process is waiting for a critical event (like I/O) and cannot be interrupted, even by signals.​ Eg mkdir makes a syscall to the kernel and during that enters this state.

  3. Stopped (T): A process is suspended, typically due to receiving a stop signal (e.g., SIGSTOP). It remains in this state until it receives a continue signal (e.g., SIGCONT).​

  4. Zombie (Z): After a process terminates, it enters the zombie state if its parent hasn't read its exit status. Zombies don't consume system resources but occupy a slot in the process table until cleared. ​

Accessing Linux Terminal and Basic Commands:

• The terminal provides a way to interact with the shell.

• Basic commands:

  • date: Displays the current date and time.

  • ls: Lists files and directories in the current directory.

  • ls -l: Lists files and directories with detailed information (permissions, owner, size, modification time).

  • mkdir <directory_name>: Creates a new directory with the specified name.

  • pwd: Prints the present working directory (the current directory you are in).

  • touch <file_name>: Creates an empty file with the specified name.

  • clear: Clears the content of the terminal screen.

  • cd <directory_name>: Changes the current directory to the specified directory.

  • cd ..: Moves up one directory level (to the parent directory).

  • cd: Returns to the user's home directory.

  • cd /: Changes to the root directory of the file system.

• The d at the beginning of ls -l output indicates a directory.

File and Directory Manipulation Commands:

• rm <file_name>: Removes (deletes) the specified file permanently (no recycle bin).

• rm <directory_name>: Will not remove a directory by default; it needs a specific option.

• rm -r <directory_name>: Removes the specified directory and all its contents recursively (use with caution). The -r flag stands for recursive.

• rmdir <directory_name>: Removes an empty directory.

Creating and Viewing File Content :

• touch: Creates a new empty file or updates the modification timestamp of an existing file.

• echo "<text>": Displays the given text on the terminal screen.

• echo "<text>" > <file_name>: Redirects the output of the echo command to the specified file, overwriting its content if it exists or creating the file if it doesn't.

• cat <file_name>: Displays the content of the specified file on the terminal.

• zcat <zip_file>: Displays the content of a zipped file without needing to unzip it first.

• head <file_name>: Displays the first 10 lines of the specified file.

• tail <file_name>: Displays the last 10 lines of the specified file.

• tail -f <file_name>: Follows the specified file in real-time, displaying new lines as they are added (useful for monitoring logs). Press Ctrl+C to stop following.

• less <file_name>: Displays the file content page by page, allowing navigation. Use spacebar to move down a page and 'q' to quit.

• more <file_name>: Similar to less, used for displaying file content page by page. It only allows scrolling forward.

• cp <source_file> <destination>: Copies files or directories from one location to another.

• cp -r <source_directory> <destination_directory>: for copying directories

• mv <source_file> <destination>: Moves or renames files and directories

• wc <file_name>: Counts the number of lines, words, and characters in a file.

  • -c, --bytes print the byte counts

  • -m, --chars print the character counts

  • -l, --lines print the newline counts

• vi <file_name>: Opens the file in the vi editor for editing.

Links (Hard and Soft):

• Soft Link (Symbolic Link):

  • Created with ln -s <source_path> <link_path>. The -s option creates a symbolic link.

  • A soft link is a pointer to the original file.

  • If the original file is deleted, the soft link becomes invalid and will likely be displayed in red.

• Hard Link:

  • Created with ln <source_path> <link_path> (without the -s option).

  • A hard link creates a new directory entry that points to the same inode (index node, data on disk) as the original file.

  • If the original file is deleted, the hard link will still allow access to the file's content.

File Content Manipulation (cut):

• cut -d <delimiter> -f <field_numbers> <file_name>: This command extracts specific columns (fields) from <file_name>, where the fields are separated by <delimiter> .

  • cut -d ',' -f 1,3 <file.csv>: This extracts the 1st and 3rd columns from file.csv where the columns are comma-separated.

• cut -b <byte(s)> <file_name>: Extracts specific bytes or ranges of bytes from each line of a file.

  • -b 1 <file>: Cuts the first byte of each line.

  • -b 1-4 <file>: Cuts bytes 1 through 4 of each line.

• <command> | tee <file_name>: Redirects output to a file while also displaying it on the terminal.

• sort <file_name> : This command sorts the lines of <file_name> in ascending order.

• diff <file1> <file2>: This compares file1 and file2, showing the lines that differ between them.

Remote Access :

• ssh -i <path_to_private_key> <username>@<public_dns>: Establishes a secure shell connection to a remote server.

  • -i <path_to_private_key>: Specifies the path to the private key file for authentication.

  • <username>: The username to log in with on the remote server.

  • <public_dns>: The public IP address or DNS name of the remote server.

• The user is prompted to confirm the connection for the first time.

• PuTTY is mentioned as a common SSH client for Windows.

Disk Usage:

• df: Displays information about disk space usage on mounted file systems.

• df -h: Displays disk space usage in a human-readable format (e.g., GB, MB).

• du .: Displays the disk usage of the current directory and its subdirectories.

• Hidden folders (starting with a .) are not shown by default with ls; use ls -a to view them.

Processes (ps, top, kill, fuser):

• ps: Provides a snapshot of the currently running processes.

• top: Displays a dynamic, real-time view of running processes, system resource usage (CPU, memory), and other system information. Press q to exit top.

• kill <PID>: Sends a signal to a process identified by its Process ID (PID).

  • kill -9 <PID>: Sends a SIGKILL signal, which forcefully terminates the process (use with caution and when other methods fail). Requires sudo for killing processes owned by other users.

• fuser <file/directory>: Identifies processes that are using the specified file or directory.

Memory Usage (free):

• free: Displays information about the system's memory usage (total, used, free, shared, buffers/cache, available).

• free -h: Displays memory usage in a human-readable format.

Running Commands in the Background (nohup):

• nohup <command> &: Executes the specified command in the background, and it will continue to run even if you log out. The output is appended to a file named nohup.out in the current directory.

• Multiple nohup commands will continue to append their output to the same nohup.out file.

System Information Commands:

• uname: Prints system information.

  • uname -a: Prints detailed system information.

  • Indicates the platform (e.g., Linux).

• uptime: Displays how long the system has been running, the number of users currently logged in, and the system load averages.

• date: Displays the current date and time.

• who: Shows a list of users currently logged in, the terminal they are using, and their login time.

• whoami: Prints the username of the current logged-in user.

• which <command>: Shows the full path of the executable for the given command. Useful for verifying if a program is installed and its location (e.g., which python3, which bash, which java).

• id: Displays the User ID (UID), Group ID (GID), and the list of groups the current user belongs to.

User Management (useradd, passwd, su, userdel):

• sudo useradd -m <username>: Adds a new user to the system. The -m option creates a home directory for the new user. Requires sudo for administrative privileges.

• sudo passwd <username>: Sets or changes the password for the specified user. Requires sudo.

• su <username>: Switches the current user to the specified username. Prompts for the password of the target user.

• exit: Logs out of the current user session and returns to the previous user.

• sudo userdel <username>: Deletes the specified user account from the system but typically leaves the user's home directory and files intact. Requires sudo.

• sudo userdel -r <username>: Deletes the specified user account and also removes the user's home directory and associated files. Requires sudo.

Group Management (groupadd, groups, cat /etc/group, groupdel, usermod -aG):

• sudo groupadd <groupname>: Creates a new group with the specified name. Requires sudo.

• groups: Displays the list of groups that the current user is a member of.

• cat /etc/group: Displays a list of all groups defined on the system. Each line typically contains the group name, password (usually x), GID, and a comma-separated list of group members.

• sudo groupdel <groupname>: Deletes the specified group. Requires sudo. A group cannot be deleted if it is the primary group of any user.

• sudo usermod -aG <groupname> <username>: Adds the specified user (<username>) to an existing group (<groupname>). The -a option ensures that the user is added to the group without removing them from any other groups they are already a member of, and -G specifies the supplementary group(s). Without the -a (append) option, using -G will replace the user's current supplementary groups with the specified ones.

• gpasswd --add username group : Both with usermod and gpasswd you can add/remove users to/from group. However, you cannot change the user's primary group with gpasswd

File Management and transfer:

• The output of ls -l displays file permissions in the format -rwxrwxrwx (for a file) or drwxrwxrwx (for a directory).

  • The first character indicates the file type (- for file, d for directory, l for link).

  • The next nine characters are grouped into three sets of three, representing permissions for the owner (user), group, and others.

  • Each set of three characters represents read (r), write (w), and execute (x) permissions. A - indicates that the permission is not granted.

• Numeric (Octal) Representation of Permissions:

  • r = 4

  • w = 2

  • x = 1

  • - = 0

  • To set permissions using numbers, you sum the values for each category (owner, group, others). For example, rwx is 4+2+1 = 7, rw- is 4+2+0 = 6, r-x is 4+0+1 = 5, etc..

  • To not mug the permission table write binary of 0 to 7 number then remember rwx format and for each entry wherever there is one fill with appropirate character from rwx , for better clarity must watch this video.

• chmod <permissions_octal> <file/directory>: Changes the permissions of the specified file or directory. Requires sudo to change permissions of files owned by other users.

  • sudo chmod 777 <directory_name>: Grants read, write, and execute permissions to the owner, group, and others for the specified directory.

  • sudo chmod 644 <file_name>: Grants read and write to the owner, and read-only to the group and others for the specified file.

  • sudo chmod 700 <file_name>: Grants read, write, and execute to the owner, and no permissions to the group and others for the specified file. Executable files may appear in green in the terminal.

File Ownership (chown, chgrp):

• sudo chown <new_owner> <file/directory>: Changes the owner of the specified file or directory to <new_owner>. Requires sudo.

• sudo chgrp <new_group> <file/directory>: Changes the group ownership of the specified file or directory to <new_group>. Requires sudo.

• umask 0002 command in Linux is used to set default permissions for new files or directories the user creates.

Compression (zip, unzip):

• sudo apt install zip: Installs the zip utility for creating zip archives.

• zip <zip_file_name> <file(s)/directory(ies)>: Compresses the specified files or directories into a zip archive named <zip_file_name>.zip.

  • zip -r <zip_file_name> <directory_name>: The -r option is used to recursively include all files and subdirectories within the specified directory.

• sudo apt install zip: Installs the zip utility for extracting files from zip archives.

• unzip <zip_file_name>: Extracts the contents of the specified zip archive into the current directory.

• cp <zip_file> <destination_directory>: Copies the zip file to the desired location before unzipping.

• cd <destination_directory>: Changes the current directory to the destination folder.

• tar <options> <archive-file> <file or directory to be archived> ‘tar’ stands for tape archive, which is used to create Archive and extract the Archive files.

  • tar cvf file.tar *.c This command creates a tar file called file.tar which is the Archive of all .c files in the current directory. Flag xvzf are x for extraction v for verbose z for gzip compression f for filename

Secure Copy (scp):

• scp -i <path_to_private_key> <source> <destination>: Securely copies files between a local and a remote system or between two remote systems.

  • -i <path_to_private_key>: Specifies the private key file for SSH authentication.

  • Copying a file from a remote server to the local machine: scp -i <path/to/your_private_key.pem> <username>@<remote_ip>:<remote_path_to_file> <local_destination_path>.

  • Copying a directory recursively from a remote server to the local machine: scp -r -i <path/to/your_private_key.pem> <username>@<remote_ip>:<remote_path_to_directory> <local_destination_path>. The -r option is for recursive copying of directories.

  • Copying a file from the local machine to a remote server: scp -i <path/to/your_private_key.pem> <local_path_to_file> <username>@<remote_ip>:<remote_destination_path>.

Remote Synchronization (rsync):

• sudo apt install rsync: Installs the rsync utility, which provides efficient file transfer and synchronization.

• rsync [OPTIONS] SOURCE DESTINATION: Command syntax for rsync.

• -a: Archive mode; preserves permissions, ownership, timestamps, symbolic links, etc..

• -v: Verbose output, showing details of the transfer.

• -z: Compresses the data during transfer to reduce bandwidth usage.

• -e 'ssh -i <path_to_private_key>': Specifies that SSH should be used as the transport for the synchronization, along with the path to the private key for authentication.

• Synchronizing a local folder with a remote folder using SSH: rsync -avz -e "ssh -i <path/to/your_private_key.pem>" <local_folder> <username>@<remote_ip>:<remote_folder>.

• rsync can synchronize in both directions; by swapping the source and destination, you can transfer files from the remote to the local machine.

Networking Commands :

• sudo apt install net-tools: Installs a collection of networking utilities, including netstat.

• netstat -tuln: Displays active network connections, listening ports, and routing tables.

  • -t: Shows TCP connections.

  • -u: Shows UDP connections.

  • -l: Shows listening sockets.

  • -n: Displays numerical addresses and ports.

• ifconfig: Displays the configuration of network interfaces (IP addresses, MAC addresses, etc.). Might be deprecated in newer systems; ip addr show is a modern alternative.

• ping <hostname/IP>: Sends ICMP echo requests to a network host to test reachability.

• host [options] name [server] The host command looks up the hostname or IP address of a remote machine by querying DNS: → hostwww.ubuntu.orgoutputwww.ubuntu.com has address 69.16.230.226

• whois [options] domain_name The whois command prints the registration of an internet domain.

• sudo apt install traceroute: Installs the traceroute utility.

• traceroute <hostname/IP>: Shows the path taken by packets to reach a destination, listing each router (hop) along the way.

• sudo apt install tracepath: Installs the tracepath utility.

• tracepath <hostname/IP>: Similar to traceroute but typically doesn't require root privileges.

• sudo apt install mtr: Installs the mtr (My Traceroute) utility.

• mtr <hostname/IP>: Combines the functionality of ping and traceroute, providing a continuous, real-time display of the network path and latency to a destination.

• uname [options] The uname command prints fundamental information about the OS, particularly the kernel

• ip [options] object command… The ip command displays and sets various aspects of your computer’s network interface Eg. ip addr show

• hostname [options] [name] The hostname command prints the name of your computer. Depending on how you have things set up, this might be the fully qualified hostname:

• telnet [hostname or IP address] [port] telnet is a network protocol used to connect to remote computers or devices, typically used for logging into remote servers. It works over TCP/IP

  • Example: telnet example.com 80 will connect to example.com on port 80 (HTTP).

  • telnet is insecure because data is transmitted in plain text. It's mostly replaced by more secure protocols like SSH.

• nslookup <hostname> is a command-line tool used to query DNS (Domain Name System) servers to obtain information about domain names, IP addresses, or DNS records.

• ss <options> is a utility used to investigate network sockets. It provides detailed information about network connections (both incoming and outgoing), listening sockets, and much more

• arp <options> (Address Resolution Protocol) command is used to view and manipulate the ARP cache, which maps IP addresses to MAC addresses in a local network. ARP is a protocol used to resolve IP addresses to hardware MAC addresses on local networks.

• nc (Netcat) is a versatile networking tool that can read and write data across network connections using TCP or UDP protocols. It is often used for port scanning, creating simple servers, and testing network connections.

• ifplugstatus is used to check the status of network interfaces on Linux. It helps determine whether an Ethernet or wireless interface is connected to a network and if a cable is plugged into an Ethernet port.

• curl <URL>: A command-line tool for transferring data with URLs. Often used for making HTTP requests to APIs.

  • curl -X GET <URL>: Explicitly specifies a GET request.

  • sudo apt install jq: Installs jq, a lightweight and flexible command-line JSON processor.

  • curl <API_URL> | jq: Pipes the JSON output from curl to jq for pretty-printing and easier readability.

• wget <URL>: A command-line utility for downloading files from the internet.

  • wget <URL_of_file>: Downloads the file located at the specified URL to the current directory.

• The route command is used to view and manipulate the IP routing table in Linux. It can show you how data packets are routed through the system and let you add or delete routing entries.

• nmap (Network Mapper) is a powerful tool for network discovery and security auditing. It can be used to scan for open ports, discover devices on a network, or even identify services and operating systems.

• iptables is used for configuring the Linux kernel's packet filtering rules. It controls the incoming and outgoing network traffic and can be used to set up firewalls, NAT, and packet filtering.

  • sudo iptables -L: Displays the current rules configured in the Linux kernel firewall (requires root privileges).

Process Monitoring and Filtering (watch, grep, awk, sed):

• watch -n <seconds> <command>: Executes the specified command every <seconds> and displays the output in a full-screen display, highlighting changes.

• grep "<pattern>" <file(s)>: Searches for lines in the specified files that contain the given <pattern>.

  • grep -i "<pattern>" <file>: Performs a case-insensitive search.

  • grep -c "<pattern>" <file>: Displays a count of the number of lines that match the pattern.

  • Piping grep with other commands: command | grep "<pattern>" filters the output of command to show only lines containing the pattern. Example: ps aux | grep ubuntu to list processes owned by the ubuntu user.

• awk '{action}' <file>: A powerful command-line utility for text processing. It works by processing files line by line, splitting each line into fields, and then performing specified actions based on patterns or conditions.

  • awk '{print $<column_number>}' <file>: Prints the specified column (field) of each line. Fields are typically separated by whitespace. $1 for the first column, $2 for the second, and so on. $0 represents the entire line.

  • awk '{if (condition) print $column}' <file>: Executes an action (e.g., print a column) only if the condition is true. Conditions can involve comparing field values (e.g., $2 >= "08:53:00").

  • awk 'BEGIN { ... } { ... } END { ... }' <file>: Structure of an awk script. BEGIN block executes before processing any lines, the middle block is executed for each line, and the END block executes after processing all lines. Example for counting occurrences: awk '{count["Info"]++} END {print "The count of Info is", count["Info"]}' <file>. NR is a built-in variable representing the current line number.

• sed 'command' <file>: The stream editor, used to perform basic text transformations on an input stream (file or output from another command).

  • sed -n '/<pattern>/p' <file>: Prints only the lines that match the specified <pattern>. The -n option suppresses default output, and /pattern/p explicitly prints matching lines.

  • sed 's/<old>/<new>/g' <file>: Substitutes all occurrences of <old> with <new> on each line. The s command performs the substitution, and g flag means global replacement on each line.

  • -e: Allows specifying multiple sed commands.

  • '1,10s/Info/Log/g': Applies the substitution only to lines 1 through 10.

  • To print the line number along with the matching line: sed -n -e '/Info/{=;p}' <file>. The = prints the line number before the matching line.

Storage Management (LVM - Logical Volume Management, EBS - Elastic Block Store):

• EBS (Elastic Block Store): Block-level storage volumes in AWS used with EC2 instances. Allows creating and attaching volumes of different sizes (e.g., 10GB, 12GB, 14GB). Root volumes are also a type of EBS volume; by default, an EC2 instance might have an 8GB root volume.

• LVM (Logical Volume Management): Provides a flexible way to manage storage in Linux by abstracting the physical disks. Key concepts:

  • Physical Volumes (PVs): Individual disk partitions or entire disks initialized for LVM. Command: sudo pvcreate /dev/xvdf /dev/xvdg.

  • Volume Group (VG): A container formed by one or more PVs, creating a pool of storage. Command: sudo vgcreate <vg_name> /dev/xvdf /dev/xvdg.

  • Logical Volumes (LVs): Virtual partitions created within a VG. LVs can be resized dynamically. Command: sudo lvcreate -L <size> -n <lv_name> <vg_name> (e.g., sudo lvcreate -L 10G -n tda_lv tda_vg).

• Mounting File Systems: After creating an LV, a file system needs to be created on it and then mounted to a directory to be accessible.

  • sudo mkfs -t ext4 /dev/<vg_name>/<lv_name>: Creates an ext4 file system on the LV.

  • sudo mkdir /mnt/<mount_point>: Creates a directory to serve as the mount point.

  • sudo mount /dev/<vg_name>/<lv_name> /mnt/<mount_point>: Mounts the LV to the specified mount point.

  • df -h: Shows mounted file systems and their usage.

  • lsblk: Lists block devices (disks, partitions, logical volumes).

  • There are both options either to make volume groups then logical volumes and mount them or directly format and mount the physical volume.

• Extending Logical Volumes: LVM allows increasing the size of an LV online.

  • sudo lvextend -L +<size> /dev/<vg_name>/<lv_name> (e.g., sudo lvextend -L +5G /dev/tda_vg/tda_lv).

  • After extending the LV, the file system on it also needs to be resized: sudo resize2fs /dev/<vg_name>/<lv_name>.

• LVM Information Commands:

  • sudo pvdisplay: Shows information about physical volumes.

  • sudo vgdisplay: Shows information about volume groups.

  • sudo lvdisplay: Shows information about logical volumes.

Conclusion:

• These are my notes made using these resources

  • Linux For DevOps In One Shot | Complete Beginners to Advanced Linux (Hindi)

  • Linux pocket guide