Automate Your AWS Workflows: Practical AWS CLI Commands and Scripts

We have seen that developers/DevOps engineers can use multiple other ways to connect to AWS apart from the AWS Console. Check out this blog for more details: Exploring Efficient Ways to Use AWS Beyond the AWS Console.

Let’s go deep into AWS CLI.

Advantages of AWS CLI:

1. Official Tool by AWS: AWS CLI is an official command-line tool provided by AWS.

2. Automation: It allows you to automate repetitive tasks via scripting, enabling efficiency and reducing human error in routine processes.

3. Cross-Platform Support: AWS CLI is available on all major platforms, including Windows, macOS, and Linux, providing flexibility for different environments.

4. Integration with DevOps Tools: It can seamlessly integrate with popular DevOps tools like Jenkins, Terraform, Docker, and Kubernetes, streamlining continuous integration/continuous deployment (CI/CD) pipelines.

Before diving into AWS CLI commands, you'll need to connect your local environment to your AWS account after installing the AWS CLI. Unlike the typical username and password authentication, we need a programmatic solution when working with the CLI. AWS provides access keys and secret access keys for this purpose, which can be found under the Security Credentials of each IAM user.

When setting this up for the first time, you’ll need to generate these keys. It's important to follow best practices and assign only the minimum required permissions to your IAM user for security purposes, as discussed earlier. This will ensure that the user has access only to the necessary AWS services.

Configuring AWS CLI:

We can configure AWS CLI by running the following command:

aws configure

You’ll be prompted to provide:

• AWS Access Key ID

• AWS Secret Access Key

• Default region (e.g., us-east-1)

• Default output format (e.g., json)

Now that the connection is ready we can look into some of the AWS CLI command.

Common AWS CLI Commands

1. EC2 Operations

• List all EC2 instances:

    aws ec2 describe-instances
  

• Start an EC2 instance:

    aws ec2 start-instances --instance-ids i-0123456789abcdef
  

• Stop an EC2 instance:

    aws ec2 stop-instances --instance-ids i-0123456789abcdef
  

2. S3 Operations

• List all S3 buckets:

    aws s3 ls
  

• Create a Bucket:

    aws s3 mb s3://mybucket
  

• Upload a file to an S3 bucket:

    aws s3 cp myfile.txt s3://mybucket/
  

• Download a file from an S3 bucket:

    aws s3 cp s3://mybucket/myfile.txt myfile.txt
  

• Sync Local Directory with S3:

    aws s3 sync ./my-folder s3://mybucket/
  

• Remove an S3 Bucket:

    aws s3 rb s3://my-bucket-name --force
  

3. IAM User Operations

• Create a new IAM user:

    aws iam create-user --user-name new_user
  

• Attach a policy to an IAM user:

    aws iam attach-user-policy --user-name new_user --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess
  

• List IAM users:

    aws iam list-users
  

These are the simple examples.Let’s look at how to filter the basic commands to get only the required result.

AWS CLI --query Option

The --query option in AWS CLI allows you to filter and format the output of AWS CLI commands using JMESPath, a powerful query language specifically designed to work with JSON data.

Key Benefits of Using --query:

1. Selective Output: Extract only the information you need, which reduces clutter.

2. Custom Formatting: You can format the results, combine fields, and more, and tailor the output to your requirements.

3. Efficiency: By filtering the output directly within the AWS CLI command, you reduce the need for additional post-processing.

Syntax:

aws <service> <operation> --query "<jmespath_expression>"

Example Use Cases:

1. Extract Specific Fields

Suppose you want to list all IAM groups but only display their names and creation dates:

aws iam list-groups --query "Groups[*].[GroupName,CreateDate]"

This command outputs only the GroupName and CreateDate fields, ignoring other details.

Suppose we need to extract the private IP’s from all ec2 instances:

aws ec2 describe-instances --query "Reservations[*].Instances[*].PrivateIpAddress"

This command filters through all EC2 instances and returns only the PrivateIpAddress of each instance.

2. Filter Based on Conditions

Let's say you want to list all running EC2 instances in a specific region:

aws ec2 describe-instances --query "Reservations[*].Instances[?State.Name=='running'].[InstanceId,InstanceType]"

This command filters out instances that are not in the "running" state and returns the InstanceId and InstanceType for the running instances.

3. Nested Data Extraction

If you're working with more complex data structures (like those returned by the AWS CLI), --query allows you to dig deep into nested objects. For example:

aws s3api list-buckets --query "Buckets[*].{Name:Name,CreationDate:CreationDate}"

This command extracts each bucket's Name and CreationDate, formatting the output as a list of dictionaries.

4. Aggregating Data

You can also perform simple aggregation. For example, to count the number of IAM users:

aws iam list-users --query "Users | length(@)"

This outputs the total number of IAM users.

Common Operators in JMESPath:

• Filtering: [*] allows you to retrieve multiple items.

• Projections: . is used to access attributes.

• Conditionals: [?condition] allows you to apply conditions.

• Aliases: {} allows you to alias the fields for better readability.

Formatting Options:

You can combine --query with the --output flag to change the output format (JSON, table, text). For example:

aws s3api list-buckets --query "Buckets[*].Name" --output text

This command lists all bucket names in plain text format.

jq Tool

jq is a powerful standalone JSON processor. It’s not integrated into AWS CLI but can be used to process the JSON output from AWS CLI commands. jq provides a lot of flexibility and power, particularly for more complex transformations and data manipulations.

Example with jq:

First, you output the raw JSON using --output json and then pipe it to jq for processing.

Extracting instance IDs using jq:

aws ec2 describe-instances --output json | jq '.Reservations[].Instances[].InstanceId'

Key Features of jq:

• Extremely powerful and flexible, especially for advanced JSON processing.

• Supports a wide range of transformations, including filtering, mapping, and aggregation.

• Can be used with any JSON output, not just from AWS CLI.

Comparison: --query vs jq

---

Examples of Using jq

1. Extracting EC2 Private IPs:

aws ec2 describe-instances --output json | jq '.Reservations[].Instances[].PrivateIpAddress'

2. Formatting Output:

List EC2 instance IDs and private IPs in a custom format:

aws ec2 describe-instances --output json | jq '.Reservations[].Instances[] | {InstanceId: .InstanceId, PrivateIpAddress: .PrivateIpAddress}'

3. Filtering by Condition:

Filter EC2 instances with a specific instance state (e.g., running):

aws ec2 describe-instances --output json | jq '.Reservations[].Instances[] | select(.State.Name == "running") | .InstanceId'

---

When to Use --query vs jq

• Use --query when:

  • You need basic filtering or extracting specific fields.

  • You want a simpler, integrated solution without extra tools.

  • You are processing data directly inside AWS CLI without complex transformations.

• Use jq when:

  • You need advanced JSON manipulation or more complex queries.

  • You want to format, filter, or aggregate data in ways not possible with --query.

  • You are working with non-AWS CLI JSON data, as jq works universally with any JSON format.

Now let’s create one script that we can implement using --query vs jq options.

Bash Script Example: List IAM Groups and Group Policies

Using --query

#!/bin/bash

# Step 1: List all IAM groups and extract their group names
aws iam list-groups --query 'Groups[].GroupName' --output text | tr '\t' '\n' > groupname_query.txt

# Step 2: Read each group name from groupname_query.txt and fetch its policies
echo "Fetching policies for each group..."

cat groupname_query.txt | while read group; do
  # Remove any unwanted newlines or spaces in group name
  group=$(echo "$group" | xargs)

  echo "Group: $group"

  # Step 3: Fetch inline policies for the group
  inline_policies=$(aws iam list-group-policies --group-name "$group" --query 'PolicyNames' --output text)

  # Step 4: Fetch managed policies attached to the group
  attached_policies=$(aws iam list-attached-group-policies --group-name "$group" --query 'AttachedPolicies[].PolicyName' --output text | tr '\t' '\n')

  # Step 5: Print the group name and attached policies
  echo "Inline Policies for group $group: $inline_policies"
  echo "Managed Policies for group $group: $attached_policies"

  # Step 6: Append to policy.txt
  echo "$group - Inline Policies: $inline_policies - Managed Policies: $attached_policies" >> policy.txt

done

echo "Policy information saved to policy.txt"

Using jq

#!/bin/bash

# Step 1: List all IAM groups and extract their group names
aws iam list-groups | jq -r '.Groups[].GroupName' > groupname.txt

# Step 2: Read each group name from groupname.txt and fetch its policies
echo "Fetching policies for each group..."

cat groupname.txt | while read group; do
  # Remove any unwanted newlines or spaces in group name
  group=$(echo "$group" | xargs)

  echo "Group: $group"

  # Step 3: Fetch inline policies for the group
  inline_policies=$(aws iam list-group-policies --group-name "$group" --output json | jq -r '.PolicyNames[]')

  # Step 4: Fetch managed policies attached to the group
  attached_policies=$(aws iam list-attached-group-policies --group-name "$group" --output json | jq -r '.AttachedPolicies[].PolicyName')

  # Step 5: Print the group name and attached policies
  echo "Inline Policies for group $group: $inline_policies"
  echo "Managed Policies for group $group: $attached_policies"

  # Step 6: Append to policy.txt
  echo "$group - Inline Policies: $inline_policies - Managed Policies: $attached_policies" >> policy.txt

done

echo "Policy information saved to policy.txt"

The AWS CLI is a powerful tool that enables you to interact with AWS services in a more efficient, automated way. By using practical commands and scripts, you can reduce manual tasks, speed up deployments, and improve your cloud operations.

Hppay Learning 😊