Postmortem Analysis

This post-mortem report details the incident that occurred on the 14th day of April 2023, the incident affected the web server and resulted in the outage of our website.

Incident Summary

The incident was reported between 2:30 PM WAT and 3:00 PM WAT. Users were having "ERR_CONNECTION_REFUSED" message when accessing the website. This problem affected 100% of all the visitors of the website. It was discovered that an incorrectly set up Nginx configuration was the primary root cause of this outage.

Timeline

2:30 PM: The problem was reported to the team when a user complained of trying to access the website.

2:35 PM: We started troubleshooting by checking the server to see if nginx was running but result showed that it was running perfectly.

2:40 PM: My team identified that the error was caused by incorrect nginx server configuration.

2:50 PM: We implemented a solution by running the script used to configure the server when setting up website the initially.

3:00 PM: We set up datadog to monitor the website's performance and to enable the team to quickly identify, resolve issues and optimize performance.

Root Cause

The root cause of this outage was that one of the engineers in the team had initially worked on the server before the incident, unknown to him that he had tampered with the port the server was listening to. To fix this error, We had to run the bash script that was written to configure our server initially.

Corrective and Preventive Measures

The following actions will be taken in the future to avoid a recurrence of similar events:

• Check and verify Nginx configurations frequently to guarantee a correct setup

• Utilize monitoring for Nginx server setups to spot any errors immediately.

• Create a procedure for inspecting Nginx configurations before deploying them. This can be done by running the command nginx -t

• Engineers should receive more instruction on Nginx setups and recommended practices.