Monitoring AWS VPC Traffic Using Flow Logs and CloudWatch

Introduction

Amazon Virtual Private Cloud (VPC) provides a logically isolated network within AWS where you can deploy and manage cloud resources securely. One of the key features of VPC is VPC Flow Logs, which helps in monitoring network traffic for security, compliance, and troubleshooting.

In this blog, we will break down the VPC Flow Logs architecture step by step to help you understand how different components interact. We will first explain VPC architecture and then move to VPC Flow Logs, its setup, and use cases.

What is Amazon VPC?

Amazon VPC is a service that allows you to create a private cloud within AWS, enabling you to launch resources such as EC2 instances, databases, and applications. It provides complete control over networking aspects, including IP addressing, routing, and security. The key components of a VPC include:

• Subnets: Logical partitions of the VPC, used to group instances.

• Route Tables: Define how traffic is directed within and outside the VPC.

• Internet Gateway: Allows resources in the VPC to communicate with the internet.

• Security Groups: Act as virtual firewalls to control inbound and outbound traffic.

• Network ACLs: Provide an additional layer of security at the subnet level.

What are VPC Flow Logs?

VPC Flow Logs is a feature that enables you to capture details about the IP traffic going to and from network interfaces within your VPC. These logs help in troubleshooting, security monitoring, and compliance auditing.

How Flow Logs Work:

1. Traffic from instances and other AWS resources is monitored at the Elastic Network Interface (ENI) level.

2. The collected flow log data is sent to Amazon CloudWatch Logs or Amazon S3.

3. The stored logs can be analyzed to detect security threats, troubleshoot network connectivity, and optimize performance.

Benefits of Using VPC Flow Logs

• Improved Security: Detect suspicious activity by monitoring network traffic.

• Network Troubleshooting: Identify issues related to connectivity, latency, and routing.

• Compliance & Auditing: Maintain logs for security audits and compliance purposes.

• Cost Optimization: Analyze unused routes and optimize resource usage.

Understanding the VPC Flow Logs Architecture

The architecture consists of the following components:

1. Clients (Users)

Users interact with AWS services and applications hosted in the VPC. The traffic they generate moves through the internet gateway and into the AWS cloud environment.

2. Internet Gateway

An internet gateway allows resources within the VPC to communicate with the internet. There are two internet gateways in the architecture – one for each VPC.

3. VPC Peering Connection

This allows communication between VPC 1 and VPC 2. It ensures private connectivity between VPCs without requiring an internet gateway or VPN.

4. Route Tables

Each VPC has a route table that defines how network traffic is directed. In the diagram:

• Route Table 1 (for VPC 1) manages traffic within that VPC.

• Route Table 2 (for VPC 2) does the same for VPC 2.

5. Network ACLs (Access Control Lists)

Network ACLs add an extra layer of security by defining rules for allowing or denying traffic at the subnet level.

6. Security Groups

Each instance is protected by a security group, which acts as a firewall controlling allowed inbound and outbound connections.

7. VPC Flow Logs

VPC Flow Logs capture network traffic and store the logs in Amazon CloudWatch. These logs provide insights into:

• Source and destination IP addresses.

• Data transfer volumes.

• Protocols used (TCP, UDP, etc.).

How to Set Up VPC Flow Logs in AWS

Follow these steps to enable VPC Flow Logs:

1. Creating two VPCs

• A public subnet hosting an EC2 instance.

• A Network ACL to control inbound/outbound traffic.

• A Route Table defines how packets move within and outside the VPC.

• An Internet Gateway to enable internet access.

Now that we understand the VPC setup, let’s launch EC2 instances, enable VPC Peering, and capture network traffic with VPC Flow Logs.

2. Launching EC2 Instances in Each VPC

To generate network traffic, we need to launch EC2 instances in both subnets.

Step 1: Create an EC2 Instance in NextWork VPC 1

1. Open AWS EC2 Dashboard → Click "Launch Instance".

2. Select Amazon Linux 2 AMI.

3. Choose Instance Type (e.g., t2.micro).

4. In Network Settings, select:

   • VPC: VPC 1

   • Subnet: Public Subnet 1

   • Security Group: Create/Assign a security group allowing SSH (port 22) and ICMP (ping).

5. Click Launch and connect via SSH.

Step 2: Create an EC2 Instance in NextWork VPC 2

Repeat the same steps but select:

• VPC: VPC 2

• Subnet: Public Subnet 2

Once both instances are running, they will act as communication endpoints, generating traffic between NextWork VPC 1 and VPC 2.

3. Enabling VPC Peering for EC2 Communication

Since the two instances are in separate VPCs, we need VPC Peering to allow direct communication.

Step 1: Create a VPC Peering Connection

1. Open AWS VPC Dashboard → Click "Peering Connections" → "Create Peering Connection".

2. Select Requester VPC → NextWork VPC 1.

3. Select Accepter VPC → NextWork VPC 2.

4. Click "Create Peering Connection" and accept the request from the Accepter VPC.

Step 2: Update Route Tables for Communication

For instance in VPC 1 and VPC 2 to communicate, and update their Route Tables:

1. Go to VPC Dashboard → Select Route Tables.

2. Edit the route for NextWork VPC 1:

   • Destination: CIDR of NextWork VPC 2

   • Target: VPC Peering Connection

3. Edit the route for NextWork VPC 2:

   • Destination: CIDR of NextWork VPC 1

   • Target: VPC Peering Connection

Now, EC2 instances can communicate through the VPC Peering Connection!

Step 3: Test Connectivity Between EC2 Instances

1. SSH into EC2 Instance in NextWork VPC 1.

2. Run:

    ping <Private IP of EC2 in VPC 2>
   

If successful, the traffic between instances will be logged by VPC Flow Logs.

4. Enabling VPC Flow Logs

To monitor network traffic, enable VPC Flow Logs.

Step 1: Navigate to the VPC Dashboard

1. Open the AWS Management Console.

2. Go to VPC Dashboard → Select your VPC.

Step 2: Create a Flow Log

1. Click "Create Flow Log".

2. Enter a name for the flow log.

Step 3: Configure Log Filters

1. Select log filtering:

   • Accept (log only allowed traffic)

   • Reject (log only denied traffic)

   • All (log all traffic)

2. Choose 10 mins for Maximum aggregation interval.

3. Choose Send to CloudWatch Logs which is the destination where we want to capture logs.

Step 3: Create a Flow Log Group in CloudWatch

1. Open the Amazon CloudWatch Console.

2. Navigate to Log groups → Click "Create log group".

3. Enter a name for your log group (e.g., VPC-FlowLogs-Group).

4. Choose Retention Settings based on how long you want to store logs.

5. Click "Create".

Step 4: Select Log Destination

1. Under Destination, select Send to CloudWatch Logs.

2. Choose the Flow Log Group you just created (VPC-FlowLogs-Group).

3. Assign an IAM Role that allows Flow Logs to write logs to CloudWatch.

4. Click "Create" to enable VPC Flow Logs.

5. Analyzing VPC Flow Logs

After enabling VPC Flow Logs, you can analyze the logs in CloudWatch Logs.

1. Open CloudWatch Console → Navigate to Log Groups.

2. Select VPC-FlowLogs-Group.

3. Search logs for network activity between EC2 instances.

Conclusion

VPC Flow Logs provide valuable insights into network traffic, helping AWS users monitor and secure their infrastructure. By following the architecture and setup steps outlined in this guide, you can effectively enable and utilize VPC Flow Logs for better network visibility and security.

Happy Learning!

Connect With Me: LinkedIn | GitHub