Postmortem Report: Security Breach on DigitalOcean Droplet

Issue Summary:

• Duration of Outage: The incident lasted approximately 3 hours, from 14:00 to 17:10 (UTC), on August 15, 2024.

• Impact: The backend API hosted on a DigitalOcean droplet was compromised, leading to a complete service outage. Since the system was still in development phase, the primary impact was on developers dependent on the API, who experienced errors when attempting to log in, with 100% of users (the developers) affected.

• Root Cause: The root cause was a misconfigured firewall allowing unrestricted access to the PostgreSQL database and the failure to change the default 'postgres' password, leading to unauthorized access by a bot, which subsequently deleted the user table and attempted to execute a malicious script.

Timeline:

• 14:00 UTC: Issue detected when a developer reported receiving a "user does not exist" error during login.

• 14:05 UTC: Initial investigation began, focusing on the application code and API logs, assuming a possible bug in the authentication process.

• 14:21 UTC: The investigation expanded to the database after confirming that the issue was not within the application logic.

• 14:30 UTC: PostgreSQL logs were reviewed, revealing that an external IP had accessed the database via port 5432.

• 14:42 UTC: Discovered that the user table had been deleted, and further logs showed attempts to download and execute a file named /tmp/kinsing.

• 15:00 UTC: The issue was escalated, and the decision was made to take the server offline for security reasons.

• 15:29 UTC: Attempts to delete the malicious /tmp/kinsing file failed as it kept reappearing after deletion.

• 16:00 UTC: A full reset of the DigitalOcean droplet was initiated, followed by restoring the server and applying enhanced security configurations.

• 17:10 UTC: Service was fully restored, and all necessary security measures were implemented.

Root Cause and Resolution:

• Root Cause: The incident was caused by two critical security oversights:

  1. Misconfigured Firewall: The firewall was configured to allow connections on port 5432 from any IP address, leaving the PostgreSQL database exposed to the internet.

  2. Default Password: The PostgreSQL database retained the default 'postgres' password, which was exploited by an automated bot to gain unauthorized access.

Once inside, the bot deleted the user table and attempted to download and execute the kinsing malware, commonly associated with cryptocurrency mining attacks. The logs indicated repeated attempts to download the malware, with error messages such as chmod: cannot access '/tmp/kinsing': No such file or directory, suggesting the bot was persistently trying to execute the script despite the file’s absence.

• Resolution: The resolution involved resetting the affected droplet, implementing a more restrictive firewall configuration, changing all default passwords, and restoring the database from a backup. Additionally, the PostgreSQL server's configuration was tightened to restrict access to trusted IP addresses only.

Corrective and Preventative Measures:

• Improvements:

  1. Enhance security training to ensure all team members understand the importance of changing default passwords and configuring firewalls correctly.

  2. Implement automated security checks for critical systems to identify vulnerabilities such as open ports and default credentials.

  3. Increase monitoring and alerting on database access patterns, especially for external connections.

• Tasks:

  1. Change all default passwords on the server and database.

  2. Reconfigure the firewall to allow connections only from trusted IPs.

  3. Set up intrusion detection and prevention systems (IDPS) on the server.

  4. Regularly audit server and database configurations for security compliance.

  5. Implement monitoring and alerts for unauthorized access attempts to the database.

  6. Educate the team on security best practices for server and database management.