Postmortem Report: Website Outage

Incident Summary: On August 8, 2024, our website experienced a three-hour outage from 2:00 PM to 5:00 PM UTC. The outage was caused by a database server failure, which resulted in the inability of the web servers to retrieve necessary data.

Image source: https://images.app.goo.gl/Lh4ZRtnnuLzi7mDJ7

Incidence Timeline:

• 2:00 PM: Website becomes unresponsive; initial investigation begins.

• 2:15 PM: Incident escalated to the database team.

• 3:00 PM: Root cause identified as a corrupted database table.

• 4:30 PM: Database restored from backup.

• 5:00 PM: Full functionality restored; website back online.

Root cause and resolution:

Root Cause: The outage was triggered by a corrupted table in the main production database. The corruption was likely due to an unhandled exception during a write operation, leading to a crash of the database service.

Resolution: The database was restored from the latest backup. After confirming data integrity and performing necessary tests, the website was brought back online.

Organizational Impact:

• Customer Impact: Users were unable to access the website during the outage, resulting in a temporary loss of service.

• Business Impact: Potential loss of revenue and customer dissatisfaction due to the unavailability of the service.

Corrective and preventative measures:

• Immediate: Review and update the backup procedures to ensure more frequent backups.

• Long-term: Implement automated monitoring and alerts for database integrity issues. Investigate the root cause of the table corruption to prevent future occurrences.

Conclusion: We apologize for the inconvenience caused and are committed to preventing similar incidents in the future. We will continue to improve our infrastructure and processes to ensure high availability and reliability of our services.