Automate the Linux and Windows servers patching using AWS system manager (SSM)

Introduction : AWS Systems Manager (SSM) provides a centralized and efficient way to automates the process of applying software patches to Amazon EC2 instances, on-premises servers, or a virtual machine (VM). SSM takes care of the patch deployment process, including staging patches, testing, and rebooting instances as needed. This helps us keep our systems secure and up to date without manual intervention on each individual server.

Advantages of Patching with SSM:

Automation : SSM allows us to automate the patching process for multiple instances. This reduces the manual effort required for patching.

Safety and Control : SSM enables us to stage patches before applying them to production instances. Using maintenance windows we can control when patches are applied, reducing the risk of patch-related downtime during critical business hours.

Visibility and Reporting: SSM provides detailed reporting and logging of patching activities.

Security and Compliance: SSM helps maintain security and compliance by ensuring that our instances are running the latest security patches and updates.

Centralized Managements: provides a centralized management console for patching. This allows you to view and manage the patch status of all your instances in one place

Rollback Capability: SSM provides rollback capabilities, allowing you to automatically revert instances to their previous state if issues are detected after patching. This helps mitigate the risk of unintended consequences.

Manual Patching :

Resource-Intensive: Manual patching can be time-consuming and resource-intensive, especially when dealing with a large number of instances spread across multiple regions.

Human Error Risk: Manual patching introduces the risk of human error, such as missing critical patches, inconsistent patching practices, or accidentally causing disruptions.

Non-Secure : For patching manually you have to share .pem file or password to access server.

Time-Consuming: Manual patching can be time-consuming, especially in environments with a large number of servers. It may also lead to variations in the patching process.

Linux Server Patch using SSM step by step : In this article, I am going to explain how to patch AWS EC2 Linux server using SSM .

Step 1 : Create tag-based resource group based on your requirement.

Step 2: Attach tags to respective EC2 instances.

Patch Group:linux2_test

Once you add the tag named as Patch Group to running instance . it will show in resource group created in first step.

Step 3 : IAM Roles : By default, AWS Systems Manager doesn’t have permission to perform actions on running Ec2 instances. You must grant access by using an AWS Identity and Access Management (IAM) instance profile.

Role Name : ec2-role-for-ssm and Policy : AmazonSSMManagedInstanceCore

Note : Don't use policy AmazonEC2RoleforSSM . This policy had deprecated. Please use AmazonSSMManagedInstanceCore policy to enable AWS Systems Manager service core functionality on EC2 instances.

1: Open the IAM console at https://console.aws.amazon.com/iam/.

2: In the navigation pane, choose Roles, and then choose Create role.

3: For Trusted entity type, choose AWS service.

4: Immediately under Use case, choose EC2, and then choose Next.

5: On the Add permissions page, do the following:

6: Use the Search field to locate the AmazonSSMManagedInstanceCore policy.

7: Select the check box next to its name

Role Created :

Note : We have to create one more customer managed inline policy and attach to same role so that when ssm scan or patch OS , output can be logged to CloudWatch.

Policy name: ec2-cloudwatch-ssm-policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Now attach the policy to above created IAM role ec2-role-for-ssm

Step 4: Attach the IAM Role ec2-role-for-ssm to instances.

Go to AWS Console -> EC2, select desired instance and hit Actions -> Instance Settings -> Attach/Replace IAM Role

Step 5 : Verify SSM Agent is working

Navigate to Managed Instances in AWS Systems Manager to check if we correctly added the IAM Role called ec2-role-for-ssm and if SSM Agent is working.

Step 6 : Now Create second IAM Role “ssm_maintenance_window_automation_service_role”

This IAM service role will be use to run ssm maintenance window task. IAM role “ssm_maintenance_window_automation_service_role” should have attached policy called “AmazonSSMAutomationRole”,"AmazonSSMMaintenanceWindowRole" and one inline policy.

On the Roles page, choose the role you just created to open the Summary page. Note the Role Name and Role ARN.

arn:aws:iam::AWS-ACCOUNTID:role/ssm_maintenance_window_automation_service_role

Now we need to add 2 customer managed policy.

SSMMaintenanceWindowPassRole and ssm-patch-sns-notification-policy

SSMMaintenanceWindowPassRole : This allows the Automation service to pass the role to other services or Systems Manager capabilities when running automations.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": [
                "arn:aws:iam::AWS-ACCOUNTID:role/ssm_maintenance_window_automation_service_role",
                "arn:aws:sns:us-west-2:AWS-ACCOUNTID:ssm-patch-notification"
            ]
        }
    ]
}

ssm-patch-sns-notification-policy : This will be use to send SNS notification.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sns:Publish"
            ],
            "Resource": "arn:aws:sns:us-west-2:AWS-ACCOUNTID:ssm-patch-notification"
        }
    ]
}

Now need to attach both customer managed policy to Automation role(ssm_maintenance_window_automation_service_role).

Final edit the Trust relationships of ssm_maintenance_window_automation_service_role.

Existing :

New :

Step 7 : Create Patchbaseline :

A patch baseline defines which patches are approved for installation on your instances. You can specify approved or rejected patches one by one. You can also create auto-approval rules to specify that certain types of updates (for example, critical updates) should be automatically approved. The rejected list overrides both the rules and the approved list.

To create a new Patch baseline please go to the AWS Console -> Systems Manager -> Patch Manager and then click create patch baseline.

1. Select Create patch baseline.

2. On the Create patch baseline page in the Provide patch baseline details section:

   • Enter a Name for your custom patch baseline, such as: custom-linux2-patch-baseline.

   • Optionally enter a description, such as: Amazon Linux 2patch baseline including security and non-security patches.

   • Select Amazon Linux 2 from the list.

3. In the Approval rules section:

   • Examine the options in the lists and ensure that Product, Classification, and Severity have values of All.

   • Leave the Auto approval delay at its default of 0 days.

   • Change the value of Compliance reporting - optional to Critical.

   • Select Add another rule.

   • In the new rule, change the value of Compliance reporting - optional to Medium.

   • Check the box under Include non-security updates to include all Amazon Linux 2 updates when patching.

   • Note If an approved patch is reported as missing, the option you select in Compliance reporting, such as Critical or Medium, determines the severity of the compliance violation reported in System Manager Compliance.

   • 

     

     

   • Step 8 : Assign Patch Group tag with custom Patch baseline:

     Currently there is no PATCH Group option is displaying in screen shot :

     Note : Once you have created the custom-patch-baseline then you have to do one time patch now activity

     To get patchGroup in option : AWS-PatchNowAssociation

   • 

After PatchNowAssociation

Now we are getting patch group option in path manger :

Assign Patch Group tag with custom Patch baseline

Now, this step is very important, we need to assign our custom Patch baseline with Patch Group tag.

Please, go to AWS Console -> Systems Manager -> Patch Manager

Find our Patch baseline for linux2, Please use search by name:custom-linux2-patch-baseline

Please remember: If we don’t specify patch baseline to patch group, AWS will use the default patch baseline for the desired OS.

Now verify compliance report

Step 9 : Create a Maintenance Window :

Now we need to create a Maintenance Window for our staging instances:

Please go to AWS Console -> Systems Manager -> Maintenance Windows and select create maintenance window.

Register targets in Maintenance Window

We want SSM to update only the instances with tag Patch Group:linux2_test

Please go to AWS Systems Manager -> Maintenance windows

Search Maintenance window by name, Patching-test-linux2-CustomBaseline, and click on the Window ID of the Patching-test-linux2-CustomBaseline

Step 10 : Create tasks in Maintenance Window

Now we need to add tasks in the Maintenance window

First Task : RUN COMMMAND

Task : Task-ScanMissingUpdatesCheckCompliance

In the Command document, select AWS-RunPatchBaseline and enter 0 as the Task Priority

AWS-RunPatchBaseline: The Run Command document for this task is AWS-RunPatchBaseline. You can choose to either scan instances or scan and patch instances. If you choose to scan instances, then Patch Manager scans each instance and generates a list of missing patches for you to review. Note that AWS-RunPatchBaseline is cross-platform and works for both Windows and Linux.

Rate control, we also need to set 1 target (this task will be executed on every instance in the target group - one by one) and stop after 1 error

Now need to add TASK 2: Automated task for creating AMI image before starting patching. This will be backup plan. If there is application or OS down after patching, we can revert changes, launching a new instance from backup AMI.

Click on Register tasks and select Register Automation task.

Now we need to create the 3rd task for Patching :

In the Command document, select AWS-RunPatchBaseline and enter 2 as the Task Priority

Testing SSM Maintenance Window

For testing purposes, I have changed the Maintenance window for every 10 minute UTC. After changing the Maintenance window, please remember to enable the Maintenance Window.

View Execution summary :

First task SCAN completed :

Second Task CreateImage Completed:

Third task Patching Completed :

Now verify Compliance report :

Verify CloudWatch Logs for details report :

SNS notification in email :

CONGRATULATIONS! We have successfully Scan the Linux Server using SSM then create an AMI image as a backup and finally patched the server and make it complaint .