How to Set Up Centralized Log Management on AWS EC2 Using Rsyslog for Efficient Monitoring

To set up remote log collection using the rsyslog service, you need to configure a centralized log collector server and multiple client EC2 machines (log senders). This setup allows you to gather logs from various servers into a single, centralized location, facilitating efficient log management and analysis across your entire infrastructure.

Spin up two EC2 servers

To begin, you'll need to launch two EC2 instances on AWS. These instances will serve distinct roles: one as the log collector (log-server) and the other as the log sender (log-client). When setting up these instances, ensure they are in the same VPC for easier communication (or can be in another VPC also), and choose an appropriate instance type based on your expected load.

Configure the Rsyslog Server (Log collector)

Update and install rsyslog

Most Ubuntu servers come with rsyslog pre-installed. However, it's good practice to verify its presence and update it to the latest version to ensure compatibility and security.

sudo apt udate && sudo apt install rsyslog

Enabling incoming logs (TCP/UDP)

Edit the main rsyslog configuration file to allow incoming log messages over TCP and UDP. This is crucial for receiving logs from remote clients.

sudo vi /etc/rsyslog.conf

Uncomment or add the following lines to enable TCP and UDP reception:

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

Save and exit the file using :wq!

Create Log Storage Template

Define a template for storing incoming logs. This helps in organising logs by host and date, making it easier to manage and analyse them.

Create or edit the configuration file:

sudo nano /etc/rsyslog.d/remote.conf

Add the following lines to specify the log storage format:

$template RemoteLogs,"/var/log/remote/%HOSTNAME%/syslog.log"
*.* ?RemoteLogs

Save and exit the file.

Create the Directory and Set Permissions

Create the directory where logs will be stored and set appropriate permissions to ensure rsyslog can write logs to it.

sudo mkdir -p /var/log/remote
sudo chown syslog:adm /var/log/remote

Configure Firewall Rules

If you're using UFW on Ubuntu, allow traffic on port 514:

sudo ufw allow 514/tcp
sudo ufw allow 514/udp
sudo ufw reload

For AWS Security Groups, navigate to EC2 > Security Groups and edit the inbound rules of the server's security group:

• Type: Custom UDP / TCP

• Port: 514

• Source: Client IPs or 0.0.0.0/0 (for testing)

Restart Rsyslog

Restart the rsyslog service to apply the changes:

Your server is now configured to collect logs from remote clients.

Configure the client EC2 machine (Log Sender)

Edit Rsyslog Configuration

Create or edit the rsyslog configuration file on the client machine to specify the log server's address.

sudo nano /etc/rsyslog.d/remote.conf

Add the following line, replacing <log-server-ip> with your log server's private IP if in the same VPC, or public IP otherwise:

*.* @@<LOG_SERVER_IP>:514     # Use @ for UDP, @@ for TCP

Save and exit the file.

Restart rsyslog

Restart the rsyslog service on the client machine to start sending logs to the server:

Verify Log Collection

To verify that logs are being sent and received correctly, use the logger utility on the client machine to generate a test log entry:

logger -p warn "test log from client server"

On the log server, check the /var/log/remote directory to see if the log entry appears:

ls /var/log/remote/
tail -f /var/log/remote/<client-hostname>/syslog.log

You should see a directory for the client machine, and within it, a log file containing the test entry.

Enhancement: Use Private IPs for Security

For enhanced security and performance, use private IP addresses for communication between the server and client if they are within the same VPC or subnet. This reduces exposure to the public internet, lowers latency, and can reduce costs.

By following these steps, you can effectively set up remote log collection using rsyslog on AWS EC2 instances, ensuring centralized log management and analysis.

Conclusion

In conclusion, remote log collection is a versatile solution that extends beyond a single server, enabling the centralized storage and management of logs from multiple servers. This capability enhances the ability to monitor and analyze system activities across an entire network, providing a comprehensive view that aids in efficient troubleshooting, security monitoring, and compliance. By leveraging a centralized log collector, organizations can streamline their log management processes and improve their overall operational efficiency.