Introduction

Amazon Web Services is the Backbone of the internet.Boto3 lets us harness the power of AWS to use as an extension of our laptops. We can build automated reports, perform sentiment analysis, send alerts and more. Let's get started!

2. What is Amazon Web Services?

Our house needs water, electricity, trash service, and cable TV. Our project needs storage, compute resources, and alerts to be successful. We buy utilities for the home. We buy AWS services for our project. AWS services are granular, so they work together or on their own. Creating an account at aws.amazon.com gives us access to the AWS Console. Anything we do here to manage our web services. To log into the console, we use the username/password we signed up with. This is the root user.

To create the key/secret for Boto3, we are going to use IAM or Identity Access Management Service. We create IAM sub-users to control access to AWS resources in our account. Credentials - or the key / secret combo are what authenticate IAM users.

1. In the console, type IAM in the Find Services Section

2. At the IAM Screen, click "Users"

3. Click "Add User"

4. Enter a username and select "programmatic access".

5. Select "Attach Existing Policies Directly" to add pre-made permission sets to our new user. Give the user the AmazonS3FullAccess policy, AmazonSNSFullAccess, AmazonRekognitionFullAccess, TranslateFullAccess, and ComprehendFullAccess. We can skip the tags screen.

Make sure we set everything correctly and create the user!

What is Boto3?

Boto3 is AWS Software Development Kit (SDK) for python. It helps the developers to create, manage and integrate services and build applications on top of AWS services like Amazon S3, EC2 and Dynamo DB.

To interact with AWS in Python, we use the Boto3 library. Let's look at an example. We import boto3. We initialize the client with an AWS service name, region, key and secret. The service name can be any of the 100+ available AWS services. The region is the geo region where our resources are located. Key and secret are like a username and password to an AWS account.

Here is an example of Boto3 client in Python:

# Generate the boto3 client for interacting with S3
s3 = boto3.client('s3', region_name='us-east-1', #you can enter any region
                        # Set up AWS credentials 
                        aws_access_key_id=AWS_KEY_ID, 
                         aws_secret_access_key=AWS_SECRET)
# List the buckets
buckets = s3.list_buckets()

# Print the buckets
print(buckets)

You can create multiple clients in Boto3 with python

# Generate the boto3 client for interacting with S3 and SNS
s3 = boto3.client('s3', region_name='us-east-1', 
                         aws_access_key_id=AWS_KEY_ID, 
                         aws_secret_access_key=AWS_SECRET)

sns = boto3.client('sns', region_name='us-east-1', 
                         aws_access_key_id=AWS_KEY_ID, 
                         aws_secret_access_key=AWS_SECRET)

# List S3 buckets and SNS topics
buckets = s3.list_buckets()
topics = sns.list_topics()

# Print out the list of SNS topics
print(topics)

Cloud storage with Amazon S3

Now, let's dive into cloud storage with AWS S3. S3 lets us put any file in the cloud and make it accessible anywhere in the world through a URL. Managing cloud storage is a key component of data pipelines. Many services we will learn will depend on an object being uploaded to S3.

The main components of S3 are Buckets and Objects. Buckets are like folders on our desktop. Objects are like files within those folders. But there's a lot of power hidden underneath. Buckets have their own permissions policies. They can be configured to act as folders for a static website They can generate logs about their own activity and write them to a different bucket. The most important thing that buckets do - they contain objects. An object can be anything - an image, a video file, CSV or a log file. There are plenty of operations we can do with objects, but for now, let's focus on what we can do with buckets. What can we do with buckets using boto3? We can create a bucket. List buckets that we have in our account. And we can delete a bucket We can only store objects in buckets, so knowing how to work with buckets is a crucial component of S3 knowledge. Let's dive into some buckets!

Here is how you create a bucket in S3:

import boto3

# Create boto3 client to S3
s3 = boto3.client('s3', region_name='us-east-1', 
                         aws_access_key_id=AWS_KEY_ID, 
                         aws_secret_access_key=AWS_SECRET)

# Create the buckets
response_staging = s3.create_bucket(Bucket='gim-staging')
response_processed = s3.create_bucket(Bucket='gim-processed')
response_test = s3.create_bucket(Bucket='gim-test')

# Print out the response
print(response_staging)

If you wish to delete a bucket, here is how you can do it:

# Delete the gim-test bucket
s3.delete_bucket(Bucket='gim-test')

# Get the list_buckets response
response = s3.list_buckets()

# Print each Buckets Name
for bucket in response['Buckets']:
    print(bucket['Name'])

In order to upload some files on the cloud here is an example code you can use in python:

# Upload filename.csv with key 2019/filename.csv
s3.upload_file(Bucket='', 
               # Set filename and key
               Filename='filename.csv', 
               Key='2019/final_report_01_01.csv')

# Get object metadata and print it
response = s3.head_object(Bucket='gid-staging', 
                       Key='path/final_report_01_01.csv')

# Print the size of the uploaded object
print(response['ContentLength'])

You can use Boto3 for many other tasks such as SNS (Simple notification service)

In SNS, we publish messages to an SNS topic, and subscribers receive those messages via email or text. Once we get to the dashboard, we can see Topics and Subscriptions on the left. Every topic has an ARN or an Amazon resource name. This is a unique id for this topic. On the bottom, we can see Subscriptions. Each one of those has a unique ID as well. We'll explore those in the next lesson. To create an SNS topic, we initialize the SNS client. This is the first time we are using a boto3 client for something other than S3. We do this by passing SNS instead of S3 as the first argument. Here is an example in code

#create a boto3 client to sns
s3 = boto3.client('sns', region_name='us-east-1', 
                         aws_access_key_id=AWS_KEY_ID, 
                         aws_secret_access_key=AWS_SECRET)

response = sns.create_topic(Name = "City_alerts")

There are several protocols in SNS, but we will only work with Email and SMS - or text messages. The endpoint is the specific phone number or email address where the message should be sent. Lastly, the status can either be confirmed or pending confirmation. Phone numbers are automatically confirmed, but for email, the user has to click a unique link to authorize the subscription. To create an SMS subscription, we initialize the boto3 SNS client. Then, we call SNS subscribe passing the TopicArn of the city_alerts topic we created in the last lesson. The protocol is SMS and the endpoint is a phone number. In our response, we get a dictionary that contains a 'SubscriptionArn' key. If we want this ARN for later use, we can grab it from the dictionary using that key.

# Subscribe a phone number to streets_critical topic
resp_sms = sns.subscribe(
  TopicArn = str_critical_arn, 
  Protocol='sms', Endpoint="+16196777733")

# Print the SubscriptionArn
print(resp_sms['SubscriptionArn'])

# Subscribe an email to streets_critical topic.
resp_email = sns.subscribe(
  TopicArn = str_critical_arn, 
  Protocol='email', Endpoint="eblock@sandiegocity.gov")

# Print the SubscriptionArn
print(resp_email['SubscriptionArn'])

Here is how you can use it to send a message

# If there are over 100 potholes, create a message
if streets_v_count > 100:
  # The message should contain the number of potholes.
  message = "There are {} potholes!".format(streets_v_count)
  # The email subject should also contain number of potholes
  subject = "Latest pothole count is {}".format(streets_v_count)

  # Publish the email to the streets_critical topic
  sns.publish(
    TopicArn = str_critical_arn,
    # Set subject and message
    Message = message,
    Subject = subject
  )

Here is a code to send a single message:

# Loop through every row in contacts
for idx, row in contacts.iterrows():

    # Publish an ad-hoc SMS to the user's phone number
    response = sns.publish(
      # Set the phone number
      PhoneNumber = str(row['Phone']),
      # The message should include the user's name
      Message = 'Hello {}'.format(row['Name'])
    )

    print(response)

Finally, wewill go beyond uploading, sharing and notifying into rekognizing using AWS Rekognition and other AWS machine learning services to recognize cats, translate language and detect sentiment. We will be capping off your learning journey by applying a real-world use case that mixes everything we’ve learned!

Boto3 follows the same pattern for all AWS services. As new services come online, we will be able to interact with them using familiar patterns. We will be using Boto3 documentation and the AWS interface to explore a new service called Rekognition.

Rekognition is computer vision API by AWS. We will focus on 2 features: Detecting objects in an image and Extracting text from images. These are common tasks extremely useful in day to day operations. But if we dig around in the docs, Rekognition does a few other things. I should also note - I'm not a computer vision expert. But to use Rekognition and boto, I don't have to be. When I learned how to use Rekognition, I clicked around on the AWS interface. Let's see how we can use rekognition for object detection. The City wants to count bicycles as they pass by a camera. Before we do anything, we must upload an image to S3. As a reminder, we initialize the S3 client, then call the upload_file method. Then, construct the boto3 rekognition client. Here is an example in code

# Use Rekognition client to detect labels
image1_response = rekog.detect_labels(
    # Specify the image as an S3Object; Return one label
    Image=image1, MaxLabels=1)

# Print the labels
print(image1_response['Labels'])

# Use Rekognition client to detect labels
image2_response = rekog.detect_labels(
    Image=image2, MaxLabels=1)

# Print the labels
print(image2_response['Labels'])

Conclusion

Congratulations! You're done! You've just learned the basics of Boto3 in Python with AWS.

you learned about Amazon Web Services, the backbone of the internet. Decoupled and granular, they work well together, or on their own. You learned how to create, list and delete buckets in S3, upload files into them and secure the files. You also know how to create automatically updated, web reports! You learned how to send messages to emails and phone numbers using SNS by creating topics and subscribing users to them. You learned how to find stray cats by detecting them in an image using Rekognition, and how to extract text from parking signs. We have only scratched the surface of what AWS can do. Data engineers regularly use AWS services such as RDS to spin up databases, Lambda to create APIs on top of their models, and EC2 to spin up virtual machines. There's even a service called Textract for extracting text and tables out of images and PDFs!. Thank you for reading this far! Hope you have a happy AWS Journey