Amazon S3 and Presigned URL : The What, Why and How.

Amazon Simple Storage Service (S3) is a highly popular cloud storage solution provided by AWS. S3 provides an object storage service offering industry-leading scalability, data availability, security, and performance. It can have multiple use cases for a variety of applications like building a data lake for AI/ML tasks, archiving huge amounts of data, distributing media files or it can even be used to host static websites.

Before starting with S3 one should be familiar with the two basic components of S3 - Buckets and Objects.

A bucket is a container for objects. An object is a file and any metadata that describes that file.

Also, an object is assigned a name which is technically called a key. The object key (or key name) uniquely identifies the object in an Amazon S3 bucket. This key is used as an identifier for this object when you want to retrieve it from S3.

By default all the buckets and objects are private. This simply means that if you have stored a media file in your bucket, then no one on the internet can access that media file or even upload any file to that private bucket. But, there can be scenarios where you want your users to perform certain actions on your private bucket. So how do you allow someone to access your private bucket? There are multiple ways to do this and here we will learn about one of these methods which is presigned URLs.

What is a presigned URL?

A presigned URL is simply a URL with a few special properties embedded specific to the Amazon S3 buckets. Using this URL anyone can access the private S3 Bucket to upload/download a file. The best part is that we can share this URL with anyone without sharing the secret credentials of our AWS account and the user will still be able to perform some action on our private bucket.

This is how an S3 presiged URL look like:

https://my-unique-bucket.s3.ap-south-1.amazonaws.com/c50f8a0e-341b-4ed8-858b-55de780b2d43?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2Z47SKWHGXY5QSUH%2F20230401%2Fap-south-1%2Fs3%2Faws4_request&X-Amz-Date=20230401T023310Z&X-Amz-Expires=3600&X-Amz-Signature=3c0c56ea31538901e9c93f8c7dd3fec463013b71b171fe3a8eefaed65f38deaa&X-Amz-SignedHeaders=host&x-id=GetObject

From the above sample URL, we can break down the important components of a presigend URL in following way -

1. Action - a pre-signed URL always has some Action (GET/POST/PUT) embedded in it which determines if you can use the URL to upload or download an object. If you see the last part of the above URL, you can find the term "GetObject" which informs that this URL can only be used to download an object.

2. Key - in Amazon S3, the key is the technical term used for the relative path of an object inside a bucket. In the case of presigned URL, it's mandatory to provide a key that will be used as a reference for the object that is being uploaded/downloaded. In the above URL, "c50f8a0e-341b-4ed8-858b-55de780b2d43" this part is the key that is already stored in S3 Bucket and we will be able to download this file using the URL.

3. Expiry duration - Any presigned URL will be live for the certain duration which is defined while generating this URL. After this expiry duration, the URL simply becomes invalid and is of no use. In the above URL, "Expires=3600" tells us that this URL will be valid for 3600 seconds.

Let's see how we can generate the presigned URLs using the latest AWS SDK (v3) for JavaScript.

const { S3Client, GetObjectCommand, PutObjectCommand } = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");

// Function to generate presigned URL for upload
async function generateSignedURLUpload () {
    const client = new S3Client({ region: "ap-south-1" }); //update region as per S3 bucket location
    const command = new PutObjectCommand({
        Bucket: "my-unique-bucket", //replace with bucket name
        Key: "sample-file.pdf", //the key name you want for this file to be uploaded
    });
    const url = await getSignedUrl(client, command, { expiresIn: 3600 });
    return url;
}

// Function to generate presigned URL for download
async function generateSignedURLDownload() {
    const client = new S3Client({ region: "ap-south-1" }); //update region as per S3 bucket location
    const command = new GetObjectCommand({
        Bucket: "my-unique-bucket", //replace with bucket name
        Key: "sample-file.pdf", //pass the key to the existing S3 object for which you need the signed URL
    });
    const url = await getSignedUrl(client, command, { expiresIn: 3600 });
    return url;
}

Use Cases

1. For example, consider you are building an app for people to upload memes and anyone using the app can view the memes shared by others. These people do not need to create an account to share memes. You are using an S3 bucket to store the memes media files and your bucket is private by default. Ideally, if you have a private bucket then these users won't be able to perform any operation in that bucket. In this scenario, you can use presigned URLs to allow your users to upload and download the media files from your bucket without worrying about authenticating and sharing credentials with the users.

2. Consider you are running a daily newspaper publishing house and you want to deliver e-papers to your paid readers. In this case, you cant share a public URL that allows downloading a pdf of the e-paper, because a public URL can be shared with almost anyone and can be misused. Here, you can use a presigned URL to allow your paid readers to download the e-paper. Since the URL will be valid for a set duration and it will allow downloading only a specific file, its misuse can be avoided.

Just like these there can be a wide range of use cases of presigned URLs where we want to focus on security as well as want to share the access to our S3 bucket with others on internet.