Automating Log Analysis for Efficient Exception Monitoring in DevOps

How to Automate Log File Processing and Exception Extraction for Faster Troubleshooting and System Monitoring

In DevOps, effective log monitoring and error detection are crucial for maintaining system health. This script automates the process of analyzing log files, specifically targeting error messages and exceptions.

It works by first calculating the current time and identifying a two-hour window, allowing the script to filter log entries based on specific timestamps. It then checks the modification time of log files to determine if they have been updated recently. Depending on this, the script extracts relevant log entries, especially focusing on exceptions, and writes them to a separate file for easier monitoring.

# Get the current time in the format "YYYY-MM-DD HH:MM:SS"
current_time=date +"%Y-%m-%d %H:%M:%S"

# Convert the current time to epoch format (seconds since 1970-01-01)
current_time_epoch=$(date -d "$current_time" +%s)

# Get the time for 2 hours ago in the same "YYYY-MM-DD HH:MM:SS" format
before_2Hours_time=date -d '2 hours ago' +"%Y-%m-%d %H:%M:%S"

# Find the most recently modified file in the directory /dir
oldfilename=find /dir -type f -exec ls -t1 {} + | head -1

# Define the path to the main log file
mainfilename="/home/labex/project/catalina.out.txt"

# Define the output file for storing exception logs
exception_file="/home/labex/project/exception.txt"

# Get the timestamp from the last line of the catalina.out log file and convert it to the desired format
last_line_timing=$(tail -n 1 catalina.out | awk '{print $1, $2}' | xargs -I{} date -d "{}" +"%Y-%m-%d %H:%M:%S")

# Get the timestamp from the first line of the catalina.out log file and convert it to the desired format
first_line_timing=$(head -n 1 catalina.out | awk '{print $1, $2}' | xargs -I{} date -d "{}" +"%Y-%m-%d %H:%M:%S")

# Get the modification time of the file and store it in epoch format
file_mod_time=$(stat -c %W "$filename")

# Calculate the time difference between the current time and the file's modification time
time_diff=$((current_time_epoch - file_mod_time))

# Define a threshold for 2 hours in seconds (2 hours * 60 minutes/hour * 60 seconds/minute)
threshold=$((2 * 60 * 60))

# Check if the file's modification time is within the threshold (i.e., modified within the last 2 hours)
if [ "$time_diff" -le "$threshold" ]; then
    # Extract logs from the old file between 2 hours ago and the last line timing and append to a temporary file
    sed -n "/$before_2Hours_time/,/$last_line_timing/p"  $oldfilename >>  /dir/tmp.txt

    # Extract logs from the main file between the first line timing and the current time and append to the same temporary file
    sed -n "/$first_line_timing/,/$current_time/p" $mainfilename >> /dir/tmp.txt

    # Filter for lines containing "Exception" and include 100 lines before and after the match, saving to the exception file
    grep -A 100 -B 100 "Exception" /dir/tmp.txt > $exception_file
else
    # If the file is not modified within the last 2 hours, extract logs from the main file between 2 hours ago and the current time
    sed -n "/$before_2Hours_time/,/$current_time/p" $mainfilename >> /dir/tmp.txt

    # Filter for lines containing "Exception" and include 100 lines before and after the match, saving to the exception file
    grep -A 100 -B 100 "Exception" /dir/tmp.txt > $exception_file
fi                              

# Clean up the temporary file
rm -rf /dir/tmp.txt