Sadservers Day3: Debugging an Nginx "Connection Refused" Error SadServers Problem

The Problem Statement

A classic issue with an Nginx web server managed by systemd where curling the curl -I 127.0.0.1:80 returns this frustrating response:

curl: (7) Failed to connect to localhost port 80: Connection refused

At first glance, this tells me that Nginx isn't running properly or isn't listening on the expected port. So, I started my investigation by checking the status of the Nginx service using systemd:

systemctl status nginx

The output revealed the service was in a failed state. More importantly, the logs indicated an error while trying to start Nginx:

nginx: [emerg] unexpected ";" in /etc/nginx/sites-enabled/default:1
nginx: configuration file /etc/nginx/nginx.conf test failed

That gave me a clear hint: there's a syntax error in the Nginx configuration file, specifically an unexpected semicolon at the very beginning of the default site configuration.

So, the next logical step was to edit that file:

sudo vi /etc/nginx/sites-enabled/default

I carefully removed the extraneous semicolon at the start of the file. Once saved, I wanted to confirm the configuration was valid overall. I ran:

sudo nginx -t

And I saw the green light:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Perfect. With confidence in the config, I restarted Nginx:

sudo systemctl restart nginx

This time, my curl command didn’t result in a connection refusal, but instead HTTP 500 Internal Server Error. This showed Nginx started but was hitting application runtime issues.

Reasons vary, so I checked the error logs to dig deeper:

sudo cat /var/log/nginx/error.log

The logs gave me some clues about resource limits being hit. Rechecking systemd logs with:

journalctl -u nginx | grep fail

I discovered repeated errors saying:

Failed to set up mount namespacing: Too many open files
Failed at step NAMESPACE spawning /usr/sbin/nginx: Too many open files

At this point, I realized systemd was limiting how many files Nginx could open, preventing the service from fully starting. This was a classic "Too many open files" error caused by an insufficient file descriptor limit set by systemd.

How I Located the Nginx systemd Service File

To fix this, I needed to adjust systemd’s limits specifically for Nginx. The first question was: Where is Nginx’s systemd service file located?

Using the status output and some common Linux conventions, I knew that custom or overridden service files typically live in /etc/systemd/system/ and since I had run the systemctl status nginx command just a short while ago, I got the confirmation I was after and proceeded to edit the file.

sudo vi /etc/systemd/system/nginx.service

Inside the [Service] section, I added a line to increase the open files limit:

LimitNOFILE=65535

But why 65535?

The number 65535 is a commonly used upper bound for file descriptors because it represents the maximum value for a 16-bit unsigned integer (2^16 - 1). It’s a high enough limit to prevent the "Too many open files" error for most typical web server workloads, including Nginx handling many connections or logging files.

After saving, I reloaded systemd configurations and restarted Nginx:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Finally, I tested again with:

curl -Is 127.0.0.1:80 | head -1

This time, it returned the expected HTTP 200 OK status:

HTTP/1.1 200 OK

Success! The default Nginx page was now properly served.

---

What I Learned

This problem reminded me that behind a simple "Connection refused" error lie multiple layers to inspect:

• Service status using systemd is the first diagnostic step.

• Config file can contain mistakes. Even a stray character can prevent startup.

• Logs are indispensable for understanding what’s going wrong after a restart and knowing where to look for logs is essential.

• System-level limits (like open files counts) can silently block services.

With this problem, walking through these diagnoses methodically helped me identify and fix the error efficiently.

---

If you face a similar Nginx "Connection refused" or HTTP 500 error on a systemd-managed server, try these steps:

1. Check service status (systemctl status nginx)

2. Fix config errors and verify syntax (nginx -t)

3. Examine logs for runtime errors (/var/log/nginx/error.log and journalctl -u nginx)

4. Adjust LimitNOFILE in the systemd service file to avoid file descriptor limits

5. Reload systemd and restart Nginx

6. Confirm by curling localhost

Taking it one step at a time and carefully interpreting errors is a reliable approach for debugging server issues like this.