Run Multiple Consul Servers on a Single Node

Do you know you can run multiple Consul servers on a single node to quickly set up a Consul cluster for testing, eliminating the need for separate VMs?

To run a Consul server on a single node, use the following command:

~ consul agent -server \
  -node server1 \
  -bootstrap \
  -data-dir $(mktemp -d) \
  -client 0.0.0.0 \
  -bind 127.0.0.1

To create a multi-node Consul cluster (e.g., a 3-node cluster), you would typically repeat this process on separate nodes, which requires additional time and resources to configure. However, running multiple servers on one node simplifies and accelerates this setup for testing purposes.

Setting Up Loopback Aliases

To overcome the challenge, you can use ifconfig to add alias IP addresses to the loopback interface (lo0). The loopback interface is a virtual network interface that allows a machine to communicate with itself, default address is 127.0.0.1. By adding aliases, you're essentially creating additional "virtual" IP addresses on the same interface, which can be used to simulate multiple separate hosts on a single machine. This is useful for testing distributed systems like Consul without needing actual separate servers.

Run the following ifconfig command to add two more loopback interfaces:

~ sudo ifconfig lo0 alias 127.0.0.2 up
~ sudo ifconfig lo0 alias 127.0.0.3 up

After running these, your machine now has three usable loopback IPs: 127.0.0.1, 127.0.0.2, and 127.0.0.3. These can be treated as different machines for networking purposes, even though they're all on the same host.

Let’s verify with ifconfig again:

 ~ ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
    options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.2 netmask 0xff000000
    inet 127.0.0.3 netmask 0xff000000
    nd6 options=201<PERFORMNUD,DAD>

Running Consul Agents as a Cluster

Now you can startup the first Consul server. This is typically the "leader" as the others join it.

~ consul agent -server \
  -node server1 \
  -bootstrap-expect 3 \
  -log-level debug \
  -data-dir $(mktemp -d) \
  -client 127.0.0.1 \
  -bind 127.0.0.1 \
  -retry-join 127.0.0.1

Then, open up other terminals and start the second 127.0.0.2 and third 127.0.0.3 servers. They discover the first one via -retry-join and connect using Consul's gossip protocol over the loopback IPs.

Start up Consul server 2:

~ consul agent -server \
  -node server2 \
  -bootstrap-expect 3 \
  -log-level debug \
  -data-dir $(mktemp -d) \
  -client 127.0.0.2 \
  -bind 127.0.0.2 \
  -retry-join 127.0.0.1

Start up Consul server 3:

~ consul agent -server \
  -node server3 \
  -bootstrap-expect 3 \
  -log-level debug \
  -data-dir $(mktemp -d) \
  -client 127.0.0.3 \
  -bind 127.0.0.3 \
  -retry-join 127.0.0.1

Verify the Leader

Once all three are up, the Consul cluster bootstraps locally on the same machine! You can verify if they can successfully elect a leader with command consul operator raft list-peers.

~ consul operator raft list-peers
Node     ID                                    Address         State     Voter  RaftProtocol  Commit Index  Trails Leader By
server1  03f64799-1e28-9d94-e3bb-4201fcff8b34  127.0.0.1:8300  leader    true   3             28            -
server2  81d858ec-123a-823e-f326-a3a5cc6e475f  127.0.0.2:8300  follower  true   3             28            0 commits
server3  4689053e-7d4e-3d31-ff1e-87ede7ef7884  127.0.0.3:8300  follower  true   3             28            0 commits

This quick setup lets you experiment with a full 3-server cluster on your laptop, and you can quickly reproduce issues and observe the behaviour among Consul leader and followers.

Clean up

When you’re done, you can remove the extra IPs with sudo ifconfig lo0 -alias 127.0.0.2 and sudo ifconfig lo0 -alias 127.0.0.3 to clean up.