Detect Abusive IPs in webservers Logs with Bash

Introduction

Monitoring NGINX and APACHE web servers logs for abusive IPs and excessive hits of bots is crucial for maintaining the security and performance of your web server.

Manually checking logs can be time-consuming and error-prone. Automating this process with a custom script can save time, reduce errors, and enhance your server’s security posture.

In this post, we’ll explore a bash script designed to automate the analysis of webserver access logs, helping you to identify abusive behavior and bots effectively especially of E-Commerce platform Websites and webservers.

Purpose and Benefits

The primary purpose of this script is to streamline the process of monitoring webservers access logs by:

Identifying Abusive IPs: Detect IP addresses with a high number of requests, indicating abusive behavior or potential attacks.
Detecting Bots: Identify and analyze requests from known bots or scrapers to help filter or block unwanted automated traffic.
Providing Detailed Reports: Generate detailed reports on abusive IPs and bots to help you make informed decisions on mitigating malicious traffic.

Benefits for Day-to-Day Activities:

Time Efficiency: Automates log analysis, saving time for other tasks.
Enhanced Security: Quickly identifies threats and abnormal behavior, allowing timely server security actions.
Accurate Reporting: Delivers detailed reports on abuse and bot activity, helping understand traffic patterns and take proactive measures.

Script Overview

Here’s a breakdown of the script and its functionality:

User Input:
- The script prompts the user to enter the path to the NGINX access log file.
- It verifies the existence of the log file before proceeding.
Abusive IP Detection:
- It analyzes IP addresses with the highest request counts to identify potential abusive behavior.
- Uses the AbuseIPDB API to check the reputation of these IPs and provides detailed reports, including abuse scores and ISP information.
Bot Detection:
- Searches for known bot patterns in the logs.
- Generates a report on bot activity, showing the number of hits from each detected bot.
Detailed Reporting:
- Lists abusive IP addresses and their hit counts.
- Provides a summary of the most hit URLs associated with these IPs.

How to Use the Script

Save the Script: Save the provided bash script to a file, for example, webserver_log_checker.sh
```
 nano webserver_log_checker.sh
```

Copy the below BASH Script:

 #!/usr/bin/bash

 api_key="e378a55f27d99f9c0b70c00925df8b4170454cfe94313e289b3a3aa29c9857c2d9a29e633506a23e"

 check_url() {
     check_url=$(grep -iw "$1" "$log_path" | awk '{ print $7 }' | sort | uniq -c | sort -k1 -n | tail -n 1)
     url=$(echo $check_url | awk '{ print $2 }')
     count=$(echo $check_url | awk '{ print $1 }')
     echo -e "\e[1;34mMost Hitting URL   :: $url\n\nURL Hitting Times      :: $count\n\e[0m"
 }

 check_ip() {
     jsondata=$(curl -ks -G https://api.abuseipdb.com/api/v2/check \
       --data-urlencode "ipAddress=$1" \
       -d maxAgeInDays=90 \
       -d verbose \
       -H "Key: $api_key" \
       -H "Accept: application/json")

     ip=$(echo $jsondata | grep -oP '(?<="ipAddress":")[^"]*')
     isp=$(echo $jsondata | grep -oP '(?<="isp":")[^"]*')
     ispubip=$(echo $jsondata | grep -oP '(?<="isPublic":)[^,]*')
     countrycode=$(echo $jsondata | grep -oP '(?<="countryCode":")[^"]*')
     iswhitelisted=$(echo $jsondata | grep -oP '(?<="isWhitelisted":)[^,]*')
     abusecore=$(echo $jsondata | grep -oP '(?<="abuseConfidenceScore":)[^,]*')

     [[ -z $abusecore ]] && abusecore=0

     color="\e[1;32m"; colort="\e[1;32m"; colorc="\e[1;32m"
     [[ $iswhitelisted == "false" ]] && color="\e[1;31m"
     [[ $abusecore -gt 20 ]] && colorc="\e[1;31m"
     [[ $abusecore -gt 0 ]] && abuse_ips+=("$ip ===> Abuse Score: $abusecore% ===> ISP : $isp")

     echo -e "\e[1;34m
 IP address   :: $ip
 Hitting Times:: $2
 ISP          :: $isp
 IsPublicIP   :: $ispubip
 Country Code :: $countrycode
 IsWhitelisted:: ${color}${iswhitelisted}\e[1;34m
 Abuse Score  :: ${colorc}${abusecore}%\e[1;34m
 \e[0m"

     check_url $ip
 }

 list_abuse_ips() {
     echo -e "\e[1;31mListing abusive IP addresses detected in logs\e[0m"
     n=1
     for ip_info in "${abuse_ips[@]}"; do
         ip_address=$(echo $ip_info | awk '{print $1}')
         hits=$(grep -c "$ip_address" "$log_path")
         echo -e "$n) $hits Hits from IP address ---> $ip_info\n"
         n=$(( n+1 ))
     done
 }

 check_bots() {
     for bots in $(grep -oh -E "\w*Bot\w*|\w*bot\w*|\w*Bytespider" "$log_path" | sort | uniq -c | sort -k1 -n | tail -10 | awk '{ print $1 ":" $2 }'); do
         count=$(echo $bots | awk -F : '{ print $1 }')
         bot=$(echo $bots | awk -F : '{ print $2 }')
         echo -e "\e[1;34m$count hits from -----> $bot\e[0m"
     done
 }

 check_ips() {
     abuse_ips=()
     for ips in $(awk '{print $1}' "$log_path" | sort | uniq -c | sort -k1 -n | tail -n 10 | awk '{ print $1 ":" $2 }'); do
         count=$(echo $ips | awk -F : '{ print $1 }')
         ip=$(echo $ips | awk -F : '{ print $2 }')
         check_ip $ip $count
     done
     if [[ ${#abuse_ips[@]} -gt 0 ]]; then
         list_abuse_ips
     else
         echo -e "\e[1;32mNo Abuse IP Address found in logs\e[0m"
     fi
 }

 choose_log_path() {
     read -p $'\e[1;32mEnter Path To Your Web Server Access Log File :: \e[0m' log_path
     if [[ -f "$log_path" ]]; then
         echo -e "\e[1;32mPreparing Reports For Logs at $log_path\e[0m"
         check_ips
         check_bots
     else
         echo -e "\e[1;31mLog file not found at the specified path. Please try again.\e[0m" && choose_log_path
     fi
 }

 main() {
     choose_log_path
 }

 main $@

Make the Script Executable:
```
 chmod +x webserver_log_checker.sh
```
Run the Script: Execute the script by providing the path to your NGINX access log file:
```
 ./webserver_log_checker.sh
```
Review the Output:
- The script will display the most hitting URLs, details about abusive IPs, and bot activity.
- Review the generated reports to take necessary actions, such as blocking abusive IPs or filtering bots.

Summary

This bash script helps system administrators and security professionals manage Linux-based web servers by automating log analysis to quickly identify and mitigate abusive traffic and bot activity, enhancing server security and performance.

Feel free to Test and Customize the script according to your specific needs and use it as a part of your server management toolkit.

Automatically Detect Abusive IPs and Bots in 'NGINX / APACHE' Logs Using Bash Script