Network traffic monitoring when your cloud provider is actively working against you

Network traffic monitoring is essential for large-scale networks and organizations to actively detect and prevent intrusions at the network level as they're happening. When your infrastructure is in the cloud, you don't always have full control over all the components. Most of the hassle of maintaining infrastructure is abstracted away from you, which means that you don't get full control sometimes.

Despite this, several cloud providers give you the option to monitor your network traffic by sending a copy of all the traffic to your monitoring infrastructure. After a full five minutes of thoroughly searching on Google about this functionality on Azure Cloud, I found that this feature was not generally available. To their credit, they do provide partner solutions that have similar functionality. But I wasn't so keen on sending all my traffic to a third party for analysis.

So I came up with a hacky solution that allows me to monitor network traffic. Despite being hacky, I think it worked pretty well. My idea was to execute tcpdump on the target system I would like to monitor over SSH and replay the capture locally on a dummy interface that my NIDS was listening on.

To start, I added a user exclusively for running tcpdump on my target.

sudo adduser packet-capture # Follow the prompts and create the user
sudo passwd -dl packet-capture # Lock the user's password

On the system that hosts my NIDS, I generated an SSH key.

ssh-keygen -t rsa -b 4096 -P '' -C 'packet-capture-user' -f ~/.ssh/id_rsa

On the target system, I added the public key to the packet-capture user's authorized_keys file, with tightened security restrictions.

sudo su - packet-capture
mkdir .ssh
echo 'from="172.26.158.165",restrict,command="tcpdump -w - -U '"'"'not ((src 172.26.158.165 and dst 172.26.158.158 and dst port 22) or (src 172.26.158.158 and src port 22 and dst 172.26.158.165))'"'"'" ssh-rsa ...' > .ssh/authorized_keys
exit

Since I had some SSH configuration on the server that requires both a public key and a password for every user to log in (AuthenticationMethods publickey,password), I had to permit this user to log in using only a public key.

echo -e 'Match User packet-capture\n\tAuthenticationMethods publickey\nMatch all' | sudo tee /etc/sshd/sshd_config

I also had to allow this user to run tcpdump without root privileges.

BINARY=$(which tcpdump)
sudo addgroup pcap
sudo chgrp pcap "$BINARY"
sudo chmod 750 "$BINARY"
sudo setcap cap_net_raw,cap_net_admin=eip "$BINARY"
sudo usermod -aG pcap packet-capture

On my NIDS, I added a dummy interface for the NIDS to listen to.

sudo ip link add dummy0 type dummy
sudo ip link set dummy0 up

Finally, I created a systemd unit file that logs into the target server, picks up the traffic using tcpdump and replays the traffic locally on the dummy interface.

cat << _EOF | sudo tee /etc/systemd/system/network-monitor.service
[Unit]
Description=Network monitor
After=network.target

[Service]
User=root
Restart=always
RestartSec=10s
ExecStart=/bin/bash -c '/usr/bin/ssh -i /root/.ssh/id_rsa -oStrictHostKeyChecking=no -oBatchMode=yes -oServerAliveInterval=5 -oServerAliveCountMax=3 packet-capture@172.26.158.158 | tcprewrite --mtu=1500 --mtu-trunc --fixcsum -i - -o - | tcpreplay -i dummy0 --pps=1000 -'

[Install]
WantedBy=multi-user.target
_EOF

And that's it! Just configure your NIDS to listen in on the dummy interface, and it should start analyzing traffic.