Load balancing multiple Rathole tunnels with Traefik HTTP and TCP routers


Introduction
This article is a continuation of Expose home server with Rathole tunnel and Traefik article, which explains how to permanently host websites from home by bypassing CGNAT. That setup works well for exposing a single home server (like a Raspberry Pi, server PC, or virtual machine), but it has a limitation: it requires one VPS (or at least one public network interface) per home server. This is because the Rathole server exclusively uses ports 80
and 443
.
But it doesn't have to be like this. We can reuse a single Rathole server for many tunnels and home servers, we just need a tool to load balance their traffic, as long as our VPS's network interface provides enough bandwidth for our websites and services.
This article explains how to achieve that using Traefik HTTP and TCP routers.
Prerequisites
- A working Rathole tunnel setup from the previous article (including a VPS and a domain name)
- More than one home server (Raspberry Pi, server PC, virtual machine, or LXC container)
Architecture overview
The problem
The main problem here is that we can't bind more than one port to ports 80
and 443
, respectively. Only one service can listen on a given port at the same time. So something like this doesn't exist:
# docker-compose.yml
services:
rathole:
image: rapiz1/rathole:v0.5.0
container_name: rathole
command: --server /config/rathole.server.toml
restart: unless-stopped
ports:
# host:container
- 2333:2333
- 80:5080,5081 # non existent syntax, can't bind two ports to a single port
- 443:5443,5444 # same
volumes:
- ./rathole.server.toml:/config/rathole.server.toml:ro
Neither the operating system nor Docker provides load balancing functionality out of the box, we need to handle it ourselves.
The solution
We need to introduce a tool for load balancing traffic between tunnels. We will use Traefik, since we already use it with the Rathole client.
For each home server, we need 2 tunnels: one for HTTP and another for HTTPS traffic:
- The tunnel for HTTP traffic will use the Traefik HTTP router as usual.
- The tunnel for HTTPS traffic is a bit more interesting and challenging. For it, we will use the Traefik TCP router running in passthrough mode, since we don't want to terminate HTTPS traffic on the VPS. Instead, we want to delegate certificate resolution to the existing Traefik instance running on the client side to preserve the current setup and architecture.
Reminder:
I already wrote about the advantage of resolving SSL certificates locally on the home server in the Architecture overview section of the previous article, but here is a quick recap:
- The home server contains its entire configuration
- The home server is tunnel-agnostic and reusable
- No coupling between the tunnel server and client, no need to maintain state or version
- Decoupled debugging
- Improved security, an additional encryption layer further down the tunnel
Traefik load balancer and Rathole server
Since we passthrough encrypted HTTPS traffic, Traefik can't read the subdomain from an HTTP request as usual. Instead, we will run the Traefik router in TCP mode, using the HostSNIRegexp matcher. This will run the router on layer 4 (TCP) instead of the usual layer 7 (HTTP).
For more in-depth info on how this works, you can read here: Server Name Indication (SNI).
Now that we understand the principle, we can get to the practical implementation.
Traefik HTTP and TCP routers
Below is the complete docker-compose.yml
that defines the Traefik TCP router and the Rathole server with 2 HTTP/HTTPS tunnel pairs for 2 home servers: pi
(OrangePi) and local
(MiniPC), in my case.
# docker-compose.yml
version: '3.8'
services:
traefik:
image: traefik:v2.9.8
container_name: traefik
restart: unless-stopped
command:
- --providers.docker=true
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.traefik.address=:8080
- --api.dashboard=true
- --api.insecure=false
- --log.level=DEBUG
- --accesslog=true
ports:
- 80:80
- 443:443
- 8080:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
- proxy
labels:
# Enable the dashboard at http://traefik.amd2.nemanjamitic.com
# http for simplicity, no acme.json file
- traefik.enable=true
- 'traefik.http.routers.traefik.rule=Host(`traefik.amd2.${SITE_HOSTNAME}`)'
- traefik.http.routers.traefik.entrypoints=web
- traefik.http.routers.traefik.service=api@internal
- traefik.http.routers.traefik.middlewares=auth
- 'traefik.http.middlewares.auth.basicauth.users=${TRAEFIK_AUTH}'
rathole:
image: rapiz1/rathole:v0.5.0
container_name: rathole
command: --server /config/rathole.server.toml
restart: unless-stopped
ports:
- 2333:2333
volumes:
- ./rathole.server.toml:/config/rathole.server.toml:ro
networks:
- proxy
labels:
### HTTP port 80 - HTTP routers ###
# pi.nemanjamitic.com, www.pi.nemanjamitic.com, *.pi.nemanjamitic.com, www.*.pi.nemanjamitic.com
# Route *.pi.nemanjamitic.com -> 5080
- 'traefik.http.routers.rathole-pi.rule=HostRegexp(`pi.${SITE_HOSTNAME}`, `www.pi.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`)'
- traefik.http.routers.rathole-pi.entrypoints=web
- traefik.http.routers.rathole-pi.service=rathole-pi
- traefik.http.services.rathole-pi.loadbalancer.server.port=5080
# Route *.local.nemanjamitic.com -> 5081
- 'traefik.http.routers.rathole-local.rule=HostRegexp(`local.${SITE_HOSTNAME}`, `www.local.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`)'
- traefik.http.routers.rathole-local.entrypoints=web
- traefik.http.routers.rathole-local.service=rathole-local
- traefik.http.services.rathole-local.loadbalancer.server.port=5081
### HTTPS port 443 with TLS passthrough - TCP routers ###
# Route *.pi.nemanjamitic.com -> 5443
- 'traefik.tcp.routers.rathole-pi-secure.rule=HostSNIRegexp(`pi.${SITE_HOSTNAME}`, `www.pi.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.pi.${SITE_HOSTNAME}`)'
- traefik.tcp.routers.rathole-pi-secure.entrypoints=websecure
- traefik.tcp.routers.rathole-pi-secure.tls.passthrough=true
- traefik.tcp.routers.rathole-pi-secure.service=rathole-pi-secure
- traefik.tcp.services.rathole-pi-secure.loadbalancer.server.port=5443
# Route *.local.nemanjamitic.com -> 5444
- 'traefik.tcp.routers.rathole-local-secure.rule=HostSNIRegexp(`local.${SITE_HOSTNAME}`, `www.local.${SITE_HOSTNAME}`, `{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`, `www.{subdomain:[a-z0-9-]+}.local.${SITE_HOSTNAME}`)'
- traefik.tcp.routers.rathole-local-secure.entrypoints=websecure
- traefik.tcp.routers.rathole-local-secure.tls.passthrough=true
- traefik.tcp.routers.rathole-local-secure.service=rathole-local-secure
- traefik.tcp.services.rathole-local-secure.loadbalancer.server.port=5444
networks:
proxy:
external: true
Let's start with the most important part: the labels
on the rathole
container that define load balancing on the two tunnels.
First, we define two HTTP routers using the HostRegexp()
matcher. It takes HTTP traffic from the entrypoint on port 80
and load balances it between two tunnels on ports 5080
and 5081
.
The second pair of labels defines a TCP router that takes traffic from the HTTPS entrypoint on port 443
, passes it through without decrypting, and load balances it between tunnels on ports 5443
and 5444
. Note that with the HostSNIRegexp()
matcher, you can't include escaped dots (.
) in the regex, you must repeat the entire domain sequence to handle the www
variant of the domain.
Also note that we use separate regex variants to match the root subdomain explicitly, e.g. pi.nemanjamitic.com
and www.pi.nemanjamitic.com
for both HTTP and TCP routers.
That's it, this is the main load balancing logic definition.
Note: Because we use HostRegexp()
and HostSNIRegexp()
on the server, you will need to use Host()
and HostSNI()
matchers for the Traefik running on the client side of the tunnel, or you will get 404
errors without additional configuration. Regex matchers on both the server and client sides seem to be too loose.
Rathole server config
Now it's just left to write the config for the Rathole server that defines 2×2 tunnels. Just make sure to use a different token and port for each tunnel.
# rathole.server.toml
[server]
bind_addr = "0.0.0.0:2333"
[server.transport]
type = "noise"
[server.transport.noise]
local_private_key = "private_key"
# separated based on token, also can NOT use same ports
# pi
[server.services.pi-traefik-http]
token = "secret_token_1"
bind_addr = "0.0.0.0:5080"
[server.services.pi-traefik-https]
token = "secret_token_1"
bind_addr = "0.0.0.0:5443"
# local
[server.services.local-traefik-http]
token = "secret_token_2"
bind_addr = "0.0.0.0:5081"
[server.services.local-traefik-https]
token = "secret_token_2"
bind_addr = "0.0.0.0:5444"
Reminder: You just need to open port 2333
in the VPS firewall for the Rathole control channel and not for the ports 5080
, 5081
, 5443
, or 5444
, because they are used by Rathole internally.
Traefik dashboard
Additionally, for the sake of debugging, we expose the Traefik dashboard using labels
on the traefik
container. To simplify the configuration and avoid handling the acme.json
file, we expose it using HTTP.
Warning: When setting the dashboard hashed password via the TRAEFIK_AUTH
environment variable, make sure to escape the $
characters properly or authentication will break. To do that, you need to use both double quotes "..."
and the escape slash '\
', as shown in the example below:
# install apache2-utils
sudo apt install apache2-utils
# hash the password
htpasswd -nb admin yourpassword
# .env"
# use BOTH "..." and \$ to escape $ properly
# this will work correctly
TRAEFIK_AUTH="admin:\$asd1\$E3lsdAo\$3Mertp57JJ4LVU.HRR0"
# this will break
TRAEFIK_AUTH="admin:$asd1$E3lsdAo$3Mertp57JJ4LVU.HRR0"
# this will also break
TRAEFIK_AUTH=admin:\$asd1\$E3lsdAo\$3Mertp57JJ4LVU.HRR0
Rathole client
The client part of the tunnel is almost the same as for a single home server. The only thing to keep in mind is to bind the specific client only to the tunnels that are meant for it, and not to all tunnels. Kind of obvious and self-explanatory, but just in case, let's be very clear and explicit.
Here, we define the rathole.client.toml
Rathole client config to bind the pi
home server to its HTTP pi-traefik-http
and HTTPS pi-traefik-https
tunnels.
# rathole.client.toml
[client]
remote_addr = "123.123.123.123:2333"
[client.transport]
type = "noise"
[client.transport.noise]
remote_public_key = "public_key"
# single client per tunnels pair
# pi
[client.services.pi-traefik-http]
token = "secret_token_1"
local_addr = "traefik:80"
[client.services.pi-traefik-https]
token = "secret_token_1"
local_addr = "traefik:443"
Similarly, here we define the rathole.client.toml
config to bind the local
home server to it's HTTP local-traefik-http
and HTTPS local-traefik-https
tunnels.
# rathole.client.toml
[client]
remote_addr = "123.123.123.123:2333"
[client.transport]
type = "noise"
[client.transport.noise]
remote_public_key = "public_key"
# single client per tunnels pair
# local
[client.services.local-traefik-http]
token = "secret_token_2"
local_addr = "traefik:80"
[client.services.local-traefik-https]
token = "secret_token_2"
local_addr = "traefik:443"
docker-compose.yml
for the Rathole client and Traefik is exactly the same as it was for a single home server. I am repeating it here just for the sake of completeness.
# docker-compose.yml
version: '3.8'
services:
rathole:
image: rapiz1/rathole:v0.5.0
container_name: rathole
command: --client /config/rathole.client.toml
restart: unless-stopped
volumes:
- ./rathole.client.toml:/config/rathole.client.toml:ro
networks:
- proxy
traefik:
image: 'traefik:v2.9.8'
container_name: traefik
restart: unless-stopped
depends_on:
- rathole
command:
# moved from static conf to pass email as env var
- '--certificatesresolvers.letsencrypt.acme.email=${TRAEFIK_LETSENCRYPT_EMAIL}'
security_opt:
- no-new-privileges:true
networks:
- proxy
# rathole will pass traffic through proxy network directly on 80 and 443
# defined in rathole.client.toml
environment:
- TRAEFIK_AUTH=${TRAEFIK_AUTH}
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./traefik-data/traefik.yml:/traefik.yml:ro
- ./traefik-data/acme.json:/acme.json
- ./traefik-data/configurations:/configurations
labels:
- 'traefik.enable=true'
- 'traefik.docker.network=proxy'
- 'traefik.http.routers.traefik-secure.entrypoints=websecure'
- 'traefik.http.routers.traefik-secure.rule=Host(`traefik.${SITE_HOSTNAME}`)'
- 'traefik.http.routers.traefik-secure.middlewares=user-auth@file'
- 'traefik.http.routers.traefik-secure.service=api@internal'
networks:
proxy:
external: true
Completed code
- Traefik load balancer and Rathole server: https://github.com/nemanjam/rathole-server
- Rathole client and local Traefik: https://github.com/nemanjam/traefik-proxy/tree/main/core
Conclusion
You can use this setup to expose as many home servers as you want, in a cost-effective and practical way, as long as your VPS has enough network bandwidth to support their traffic. It can bring your homelab to another level.
What tool and method did you use to expose your home servers to the internet? Do you like this approach, are you willing to give it a try? Let me know in the comments.
Happy self-hosting.
References
- Traefik
v2.9
HostRegexp
reference: https://doc.traefik.io/traefik/v2.9/routing/routers/#rule - Traefik
v2.9
HostSNIRegexp
reference: https://doc.traefik.io/traefik/v2.9/routing/routers/#rule_1 - TLS Server Name Indication (SNI), Wikipedia https://en.wikipedia.org/wiki/Server_Name_Indication
Subscribe to my newsletter
Read articles from Nemanja Mitic directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
