Logs are an essential part of daily lives for a system administrator. They offer a window into the system's events, including error messages, security warnings, and critical events that need immediate attention. However, analyzing logs manually can be time-consuming and error-prone. So, why not automate it with a bash script?

In today’s task, we’ll create a Log Analyzer and Report Generator that automatically processes log files, counts errors, extracts critical events, and generates a summary report.

Challenge Overview

Objective

The task is to write a bash script that:

Takes a log file path as a command-line argument.
Counts the total number of error messages.
Identifies critical events.
Generates a summary report that includes:
- Date of analysis.
- Total lines processed.
- Total error count.
- Top 5 error messages with their occurrence count.
- List of critical events with their line numbers.

We’ll also add an optional enhancement to archive the log file after processing.

Tools Used

To accomplish this, we’ll use:

grep: For searching specific keywords.
awk: For text processing and filtering.
sort and uniq: For counting occurrences of specific messages.

The Script

Here's the script that automates the log analysis and report generation:

#!/bin/bash

# Check if the log file path is passed as an argument
if [ $# -ne 1 ]; then
    echo "The script should be executed as follows: ./log_analyzer.sh <log_file>"
    exit 1
fi

LOG_FILE=$1

# Check if the log file exists
if [ ! -f "$LOG_FILE" ]; then
    echo "Error: Log file $LOG_FILE does not exist."
    exit 1
fi

# Variables for report
DATE=$(date +"%Y-%m-%d %H:%M:%S")
TOTAL_LINES=$(wc -l < "$LOG_FILE")
ERROR_COUNT=$(grep -i -c "ERROR\|Failed" "$LOG_FILE")
CRITICAL_EVENTS=$(grep -in "CRITICAL" "$LOG_FILE")

# Generate the report file
REPORT_FILE="log_analysis_report_$(date +"%Y-%m-%d_%H-%M-%S").txt"
echo "Log Analysis Report - $DATE" > $REPORT_FILE
echo "Log File: $LOG_FILE" >> $REPORT_FILE
echo "Total lines processed: $TOTAL_LINES" >> $REPORT_FILE
echo "Total error count: $ERROR_COUNT" >> $REPORT_FILE

# Extract and count top 5 error messages
echo -e "\nTop 5 Error Messages:" >> $REPORT_FILE
grep -i "ERROR\|Failed" "$LOG_FILE" | awk -F: '{print $NF}' | sort | uniq -c | sort -nr | head -5 >> $REPORT_FILE

# List critical events with line numbers
echo -e "\nCritical Events:" >> $REPORT_FILE
if [ -z "$CRITICAL_EVENTS" ]; then
    echo "No critical events found." >> $REPORT_FILE
else
    echo "$CRITICAL_EVENTS" >> $REPORT_FILE
fi

echo "Report saved to $REPORT_FILE"

# Optional: Move processed log file to archive
ARCHIVE_DIR="logs_archive"
mkdir -p $ARCHIVE_DIR
mv $LOG_FILE $ARCHIVE_DIR/
echo "Log file moved to archive: $ARCHIVE_DIR/$(basename $LOG_FILE)"

Explanation of the Script

Argument Check: The script first checks whether a log file path has been passed as a command-line argument. If not, it shows the correct usage and exits.
File Existence Check: It then checks if the specified log file exists. If the file is missing, it exits with an error.
Error and Critical Event Analysis: Using grep, the script searches for errors and critical events in the log file. We count error occurrences (ERROR or Failed) and print lines containing the word “CRITICAL” along with their line numbers.
Top 5 Error Messages: The script extracts error messages and uses sort and uniq to identify the most common ones.
Report Generation: All the analyzed data is saved to a report file, including total lines processed, error count, top error messages, and critical events.
Optional Log Archiving: After the analysis is complete, the processed log file is moved to an archive directory.

How to Run the Script

Once you’ve written the script, save it as log_analyzer.sh and give it executable permissions:

chmod +x log_analyzer.sh

Then, execute the script with your log file as an argument:

./log_analyzer.sh /path/to/logfile.log

The script will generate a summary report and move the processed log file to the archive directory.

Example Output

Here's an example of what the output might look like:

ubuntu@ip-172-31-23-226:~/day10$ ./log_analyzer.sh sample_log.log
Report saved to log_analysis_report_2024-10-24_11-05-53.txt
Log file moved to archive: logs_archive/sample_log.log
ubuntu@ip-172-31-23-226:~/day10$ cat log_analysis_report_2024-10-24_11-05-53.txt
Log Analysis Report - 2024-10-24 11:05:53
Log File: sample_log.log
Total lines processed: 2000
Total error count: 305

Top 5 Error Messages:
     99 QuorumCnxManager$RecvWorker@762] - Connection broken for id 188978561024, my id = 3, error = 
     96 QuorumCnxManager$RecvWorker@762] - Connection broken for id 188978561024, my id = 2, error = 
     94 QuorumCnxManager$RecvWorker@762] - Connection broken for id 188978561024, my id = 1, error = 
     12 LearnerHandler@562] - Unexpected exception causing shutdown while sock still open
      1 QuorumCnxManager$RecvWorker@762] - Connection broken for id 3, my id = 1, error = 

Critical Events:
No critical events found.
ubuntu@ip-172-31-23-226:~/day10$ ls logs_archive
sample_log.log

Optional Enhancement: Automating the Script with Cron

You can schedule the script to run daily using crontab:

0 0 * * * /path/to/log_analyzer.sh /path/to/logfile.log

This will execute the script every day at midnight and generate a fresh report automatically.

Conclusion

Log analysis is an essential part of system administration, and automating the process saves time and reduces human error. With this bash script, you can quickly analyze logs, identify critical events, and generate a summary report, making it easier to keep track of system health.

Feel free to try out the script, enhance it further, and share your thoughts!

Log Analyzer and Report Generator

Table of contents