SSL Termination With HAproxy: The Pain of An Error

Introduction

If you are reading this article, chances are you know what SSL termination is, you ran into the same problem I did, or you might be one curious individual looking to find out what all of this is about. And how I solved my "so-called error". Ah! Have the floor seat, gladly.

SSL termination is a process used in network security and management where the Secure Sockets Layer (SSL) encryption or decryption is handled by a specific device or server (in my case, HAproxy) rather than the application server itself.

Configuring SSL termination on a load balancer like HAProxy is crucial for managing encrypted traffic efficiently. However, during this setup, you might encounter various errors that can be challenging to troubleshoot. In this article, I'll discuss one such error—No private key found...—and how I resolved it.

The Problem

I installed HAProxy on my server and configured it according to my specific needs. Which were:

Frontend configuration for HTTP

  frontend noblet-frontend-http
      bind *:80
      mode http
      default_backend noblet-backend

Backend configuration — using the roundrobin algorithm

  backend noblet-backend
      balance roundrobin
      server web-one 100.25.38.194:80 check
      server web-two 34.224.3.62:80 check

After this, I installed Certbot and used it to obtain a free SSL certificate from Let’s Encrypt. While setting up SSL termination on my HAProxy load balancer, I configured HAProxy to use the SSL certificate. This involves modifying the HAProxy configuration file to specify the location of the SSL certificate and to enable SSL termination.

Frontend configuration for HTTPs

frontend noblet-frontend-https
    bind *:443 ssl crt /etc/letsencrypt/live/www.noblet.tech/fullchain.pem
    http-request set-header X-Forwarded-Proto https
    redirect scheme https if !{ ssl_fc }
    default_backend noblet-backend

Then, the error:

Despite having the private key file in the same directory as my SSL and CA certificates, HAProxy couldn't locate the private key, halting my progress.

Investigating the Issue

After encountering the error, I checked my configuration to ensure all file paths were correct. The directory contained the following files:

root@530350-lb-01:/etc/letsencrypt/live/www.noblet.tech# ls
README  cert.pem  chain.pem  fullchain.pem  privkey.pem

The error persisted despite the files being present and the path being correct. This kept me on for hours, reading, researching, and trying out different things. Looking at the error shot above, it looked as if HAProxy expected a file with the extension .key. So, I changed the file extension; same error.

I checked the file permissions to ensure the HAProxy user has the permissions to read the files and checked if the private key file might be corrupted. Everything seemed alright, yet the error still persisted.

Before this, I read in a book that HAProxy automatically finds the files and reads them, provided they are in the same directory.

Well, served the directory path to HAProxy and that didn't work:

bind *:443 ssl crt /etc/letsencrypt/live/www.noblet.tech/

The Solution

After some research and experimentation, I discovered that HAProxy requires the SSL certificate and the private key to be in the same file so it can read them together. Alas! My pain has ended.

Why didn't I think of this earlier? Humbled but I learned something and had fun while at it.

Here’s how I resolved the issue:

Combine the SSL Certificate and Private Key:

I created a single file certandprivkey.pem containing both the SSL certificate and the private key. This can be done using a simple command or manually combining the files. Here’s how I did it using the command line:
```
 cat fullchain.pem privkey.pem certandprivkey.pem
```
This command concatenates the contents of fullchain.pem (contains the server certificate and intermediate certificate) and privkey.pem (contains the private key) into a new file certandprivkey.pem.

Update HAProxy Configuration:

I updated the HAProxy configuration to use the combined file.

File path:

 /etc/letsencrypt/live/www.noblet.tech/certandprivkey.pem

Frontend Configuration for HTTPS (noblet-frontend-https):

 frontend noblet-frontend-https
     bind *:443 ssl crt /etc/letsencrypt/live/www.noblet.tech/certandprivkey.pem
     http-request set-header X-Forwarded-Proto https
     redirect scheme https if !{ ssl_fc }
     default_backend noblet-backend

Configuration Check:

Before starting HAProxy, it’s advisable to check the configuration file for syntax errors.
```
 root@530350-lb-01: sudo haproxy -c -f /etc/haproxy/haproxy.cfg
 Configuration file is valid
```
Restart HAProxy:

I restarted the HAProxy service to apply the changes:
```
 sudo systemctl restart haproxy
```

Finally, check the status of HAProxy. Here, you can see that the HAProxy service is active and running as expected. Isn't that something? Ha!

root@530350-lb-01: sudo systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-07-02 17:29:08 UTC; 46s ago
       Docs: man:haproxy(1)
             file:/usr/share/doc/haproxy/configuration.txt.gz
   Main PID: 292929 (haproxy)
     Status: "Ready."
      Tasks: 2 (limit: 1143)
     Memory: 45.1M
     CGroup: /system.slice/haproxy.service
             ├─292929 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
             └─292931 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Jul 02 17:29:07 530350-lb-01 systemd[1]: Starting HAProxy Load Balancer...
Jul 02 17:29:08 530350-lb-01 haproxy[292929]: [NOTICE]   (292929) : New worker (292931) forked
Jul 02 17:29:08 530350-lb-01 systemd[1]: Started HAProxy Load Balancer.
Jul 02 17:29:08 530350-lb-01 haproxy[292929]: [NOTICE]   (292929) : Loading success.

Testing HAProxy Load Balancer and SSL Termination

There are three (3) servers:

web-one - web server 1
web-two - web server 2
lb-01 - load balancer

In the HAProxy configuration, I used the roundrobin algorithm. The roundrobin algorithm is a simple and widely used load-balancing technique that distributes client requests evenly across a pool of servers. In the context of HAProxy, it means that each incoming request is assigned to the next server in the list, cycling back to the first server once it reaches the end of the list. This helps ensure that no single server is overwhelmed with too many requests while others remain idle.

Rounrobin and SSL Termination in Action:

Requests are made to https://www.noblet.tech; note that I am making the request to https and not http. Each response that comes back is served by a different server, demonstrating the roundrobin algorithm in action: the first request is served by web-02 x-served-by: web-two, second is served by web-01 x-served-by: web-one indicating that the load balancer is sharing the workload between the two servers.

noblet@NOBLET:~$ curl -SI https://www.noblet.tech
HTTP/2 200
server: nginx/1.18.0 (Ubuntu)
date: Tue, 02 Jul 2024 18:23:46 GMT
content-type: text/html
content-length: 612
last-modified: Tue, 21 Apr 2020 14:09:01 GMT
etag: "5e9efe7d-264"
x-served-by: web-two
accept-ranges: bytes

noblet@NOBLET:~$ curl -SI https://www.noblet.tech
HTTP/2 200
server: nginx/1.18.0 (Ubuntu)
date: Tue, 02 Jul 2024 18:23:49 GMT
content-type: text/html
content-length: 612
last-modified: Tue, 21 Apr 2020 14:09:01 GMT
etag: "5e9efe7d-264"
x-served-by: web-one
accept-ranges: bytes

Conclusion

Combining the SSL certificate and private key into a single file resolved the "No Private Key found..." error in my HAProxy configuration. This approach simplifies the SSL termination setup and ensures that HAProxy can properly access the necessary cryptographic components.

By sharing this solution, I hope to help others who might face similar issues during their SSL termination configuration with HAProxy, especially my backend and devops peers at HNG Internship.

Congratulations on getting to the end of this article.

Do you find this resource helpful? Have a question or discovered a mistake or typo...? Please leave your feedback in the comments.

Don't forget to share this resource with others who may benefit from it as well. Follow me for more.