In this article, we are going to look into the powerful commands for text processing and file manipulation with detailed syntax explanations to have a better understanding of the concept.

Without further ado let's dive in and look at the commands sed, awk, find

SED Command

The sed command stands for Stream Editor which is one of the text processing tools used to perform various operations on text files like find & replace (substitution), insertion & deletion of lines.

The basic syntax of the sed command is

sed [options] [command] [file]

options - specifies the behavior of the sed command
command - operation the sed command should perform
file - name of the file the sed command should operate on

Find and Replace (Substitution)

1. Replace the first occurrence of each line

sed 's/<old_text>/<new_text>/' <fileName>

s stands for substitute
/ is the delimiter. We can use other delimiters like ., space as well.
this replaces the first occurrence of the old text with new text in each line and displays the output on the screen leaving the original file unmodified

2. Replace all occurrences

sed 's/<old_text>/<new_text>/g' <fileName>

g stands for global replacement
this replaces all occurrences of the old text with new text in each line and displays the output on the screen leaving the original file unmodified

3. Replace all occurrences & modify the original file

sed -i 's/<old_text>/<new_text>/g' <fileName>

i stands for in-place (edits the original file)
this replaces all occurrences of the old text with new text and updates the original file as well

Insertion

1. Insert text at a specific line number

sed '<lineNumber> i\<text>' <fileName>

i stands for insert
this inserts the text at the specific line number of the file mentioned and leaves the original file unaffected
Examples
- sed '2 i\Vish - DevOps Engineer' samplefile.txt > outputfile.txt - Inserts the text "Vish - DevOps Engineer" at the 2nd line of the file and writes it back to the output file

Deletion

1. Delete based on line number

sed '<lineNumber>d' <fileName>

d stands for delete
deletes the exact line number from the file and displays the output on the terminal
Examples
- sed '2d' samplefile.txt - deletes the 2nd line of the file mentioned
- sed '2,5d' samplefile.txt - deletes the range of lines from 2nd to 5th of the file mentioned

2. Delete based on text/regex

sed '/<text_or_regex>/d' <fileName>

deletes the exact line containing the text/regex of the file mentioned and displays the output on the terminal leaving the file unmodified
Examples
- sed '/DevOps/d' samplefile.txt - delete the line containing "DevOps" and prints the other lines in the terminal leaving the file unmodified

Text Filtering

1. Print lines that match a text/pattern

sed -n '/<search_text>/p' <file>

p stands for print which is used to display the matches lines
n used to print only matched lines and suppresses non-matching lines
Examples
- sed -n '/docker/p' samplefile.txt - displays only the lines in the terminal where the search text docker is present

2. Conditional replacement

sed '/<conditional_text>/s/<old_text>/<new_text>/' <file>

s stands for substitute
replaces the old text with the new text where the line contains the conditional text
Examples
- sed '/fruit/s/apple/orange/' samplefile.txt - replaces the apple text with orange where the line contains only fruit text and displays the changes on the terminal
- sed '/^food/s/apple/orange/' filename - replaces the apple text with orange where the line only starts with food and displays the changes on the terminal

AWK Command

awk is another powerful text processing tool used to process and manipulate structured data such as log files and command outputs. It is used to print specific column values for us as required

The basic syntax of the awk command is

awk 'pattern { action }' <file>

pattern - specifies the pattern or condition that should match
action - specifies the action to be performed when the pattern is matched
file - name of the file the awk command should operate on

1. Print specific columns containing a text/pattern

awk '/text_or_pattern/ { action }' <file>

filters the lines based on the pattern or text on the file mentioned and does the appropriate action mentioned.
Examples
- awk '/TRACE/ { print $1, $5 } samplefile.txt' - the command searches for lines in the file samplefile.txt that contain the word "TRACE" and when a match is found, the print statement displays the 1st & 5th column on the terminal. Perfect use cases for awk command is for log file filtering.

2. Print specific columns with input from the command

<linux_command> | awk { print $columnNumber, $columnNumber }

the output of the Linux command is fed as the input to the awk command using the | pipe filter
here , is used as a separator where column data is printed with a space in the terminal
Examples
- df -h | awk '{ print $1, $4 }' - prints the filesystem ($1) & the total available space ($4)
- ls -al | awk '{ print $9 }' - prints the filename of the directory ($9)

3. Print specific columns from a file

awk '{ print $columnNumber, $columnNumber }' <file>

displays the mentioned columns of the structured data file in the terminal
Examples
- awk { print $2, $4 } samplefile.txt - displays the 2nd & 4th column of the file

Note: Awk is a programming language by itself, allowing you to write more complex scripts for text-processing tasks.

Find Command

find command is used to search for files and folders in a specific directory based on various criteria.

It can locate files based on names, types, sizes, permissions, timestamps and other attributes

The basic syntax of the find command is

find <target_directory_path> -<attribute> <fileName_or_directoryName>

1. Find files by name

find <target_directory_path> -name <fileName>

searches for files with the name attribute in the target directory & its subdirectories and displays the output in the terminal
Examples
- find / -name samplefile.txt - searches for the samplefile.txt in the root directory (/ stands for root directory)

2. Find files by type

find <target_directory_path> -type f

f stands for file
locates regular files

find <target_directory_path> -type d

d stands for directory
locates directories

find <target_directory_path> -type l -name "symbolic_link_name"

l stands for symbolic link
locates the symbolic link with a specific name

3. Find files by size

find <target_directory_path> -size <+/-size>

searches for files based on size
Examples
- find . -size +1G - searches for files which are greater than 1G (+ indicates larger/greater than)
- find . -size -1G - searches for files which are lesser than 1G (- indicates smaller/lesser than)

T - terabytes G - gigabytes, M - megabytes, k - kilobytes, c - bytes

4. Find files by permissions

find <target_directory_path> -perm <octalNumber>

searches for files based on octal number permissions
Examples
- find /home/ubuntu/vish -perm 744 - displays all files which are having permissions with 744 in the mentioned directory

5. Find files by specific user

find <target_directory_path> -user <userName>

searches and displays files whose owner is as mentioned by the user name
Examples
- find /home/ubuntu/vish -user vish - displays all files which has the owner name vish inside the mentioned directory

Extras

Locate Command

The locate command in Linux is used to quickly search for files and directories on the system based on their names. It relies on a pre-built database called the locatedb for faster searching

Initially perform updatedb to refresh the locatedb to have uptodate information

locate <fileName>

displays the matching file name path in the terminal
Examples
- locate samplefile.txt - locates & displays the absolute path of samplefile.txt
- locate "*.txt" - locates & displays all files ending with .txt
- locate -i samplefile.txt - case insensitive search and displays the absolute path of samplefile.txt (i - case insensitive)

Superb, you should be now proficient in sed, awk, find commands. Hope I was able to make a good understanding of the concept. Thank you.

Please let me know in the comment section about any corrections/mistakes, am open to improvements and suggestions.

Please do follow me for more such content related to DevOps world.

Cheers, Vish. Happy Learning!!!

Demystifying Sed, Awk and Find Commands

Table of contents