ebs volume monitoring

Scenario:

Suppose you have a requirement that you need to get an notification mail if your EBS volume is 80% utilized. Cloudwatch doesn’t provide the built-in metrics for EBS Volume utilization so, you need to install an agent that will publish the custom metric on the cloudwatch and you can set an alarm based on that metric.

Architecture Diagram:

Procedure:

Step1: Launch the EC2-Instance

Step2: Copy the Public-IP address of the Instance.

Step3: Connect the server using ssh key

ssh -i /path/<Keyfile.pem> ec2-user@<public Ip>
#example:
ssh -i max-demo-key.pem ec2-user@18.207.190.7

Step3: Install and configure the cloudwatch agent

you can do it manually or configure with with a custom script.

make a script install.sh and place the following code.

#!/bin/bash

# Update package list
echo "Updating system packages..."
sudo yum update -y || sudo apt-get update -y

# Install CloudWatch Agent
echo "Installing Amazon CloudWatch Agent..."
sudo yum install -y amazon-cloudwatch-agent || sudo apt-get install -y amazon-cloudwatch-agent

# Create CloudWatch Agent Configuration
echo "Creating CloudWatch Agent Configuration for EBS volume monitoring..."
CONFIG_FILE="/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json"

sudo tee $CONFIG_FILE > /dev/null <<EOF
{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "root"
    },
    "metrics": {
        "metrics_collected": {
            "cpu": {
                "measurement": [
                    "cpu_usage_idle",
                    "cpu_usage_iowait",
                    "cpu_usage_user",
                    "cpu_usage_system"
                ],
                "metrics_collection_interval": 60,
                "totalcpu": false
            },
            "disk": {
                "measurement": [
                    "used_percent",
                    "inodes_free"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "diskio": {
                "measurement": [
                    "io_time",
                    "write_bytes",
                    "read_bytes",
                    "writes",
                    "reads"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ],
                "metrics_collection_interval": 60
            },
            "netstat": {
                "measurement": [
                    "tcp_established",
                    "tcp_time_wait"
                ],
                "metrics_collection_interval": 60
            },
            "swap": {
                "measurement": [
                    "swap_used_percent"
                ],
                "metrics_collection_interval": 60
            },
            "custom_metrics": {
                "measurement": [
                    "CustomMetricExample"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ],
                "default": {
                    "CustomMetricExample": 3
                }
            }
        }
    }
}

EOF

Step4: make the script executable and run the script

chmod +x install.sh
./install.sh

Step5: Enable and Start CloudWatch Agent

sudo systemctl enable amazon-cloudwatch-agent
sudo systemctl start amazon-cloudwatch-agent
#verify the agent is running, if required restart the agent
sudo systemctl status amazon-cloudwatch-agent --no-pager

Step6: Create a role for agent to put metric about the resources :

Search for IAM > Select Policy > Create Policy

CloudWatchAgentPolicy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "cloudwatch:DescribeAlarms",
                "cloudwatch:GetDashboard",
                "cloudwatch:ListMetrics",
                "cloudwatch:ListDashboards",
                "cloudwatch:PutMetricAlarm",
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:DescribeLogGroups",
                "logs:PutRetentionPolicy"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeTags",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumeStatus",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeRegions"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Create Policy.

Now
Select Role > Create New Role
Trusted entity type > Select Custom trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Click on Next
Select the Permission you have created Recently i.e CloudWatchAgentPolicy

Click on Next
Enter the Role Name
Click on Create Role

Step6: Attach the Role to the Instance

On the Instances
Select the Action > choose Security > Click on Modify IAM role

Choose the CloudWatchAgentInstanceProfile

Step7: Restart the Agent

sudo systemctl restart amazon-cloudwatch-agent

view the logs of the agent

sudo journalctl -u amazon-cloudwatch-agent --no-pager -n 50
sudo cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Step 9: Create a SNS Topic and make Subscription using required email address.

Click on Create Topic
Copy the Arn value

Step9: Command to create a cloudwatch alarm.

Change the required value also, CloudWatch metric parameter must match the dimensions.

Example: <Private_IP> and <SNS_ARN_Topic>

aws cloudwatch put-metric-alarm \
  --alarm-name "Volume 80% Utilization" \
  --metric-name "disk_used_percent" \
  --namespace "CWAgent" \
  --statistic "Average" \
  --dimensions Name=device,Value=xvda1 Name=fstype,Value=xfs Name=host,Value=ip-<Private-IP>.ec2.internal Name=path,Value=/ \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --period 300 \
  --alarm-actions <SNS_ARN_Topic> \
  --region us-east-1

Or manually set an alarm.

Now,

Go to the CloudWatch
Select All Alarms.
Copy the ARN of the CloudWatch Alarm.

Step10: Attach the Access policy to the SNS Topic

CloudWatch need permission to publish the message,

Click on Edit in SNS Topic
Under Access Policy replace the policy and change <SNS ARN>

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Service": "cloudwatch.amazonaws.com"
            },
            "Action": "SNS:Publish",
            "Resource": "<SNS ARN>",
            "Condition": {
              "ArnLike": {
                "aws:SourceArn": "CLOUDWATCH ARN"
              }
            }
          }
        ]
      }

Step11: Restart the Agent and check the metric list.

#restart t-status
sudo systemctl restart amazon-cloudwatch-agent
#checking the metric being pushed by the CWAgent
aws cloudwatch list-metrics --namespace "CWAgent" --metric-name "disk_used_percent" --region us-east-1

Step12: Creating Subscriptions for your SNS Topic

Go the SNS TOPIC
Create Subscriptions
Protocal: Email
EndPoint: “Enter Your Email address”

Open your Email and Confirm Subscription

Step 13: Triggering the Event by Manually utilizing the EBS Volume.

Creating a testfile.img with size of 5GB

sudo fallocate -l 5G /testfile.img
#view  the storage of the system 
df -h

Congratulation!!!

You did a great job. you have now successfully demonstrated monitoring EBS volume using CloudWatch agent and setup an alarm when the threshold met.

EBS Volume Monitoring through CloudWatchAgent and Setting an Alarm