What are presigned urls?

Presigned urls as the name suggest are presigned. They have their signature / token with them.

When you want to upload a file to some object storage service like AWS S3 or GCS, there are two ways you can upload files

1. Through backend, using AWS / GCP standard libraries.

2. Through presigned urls

Why would you do presigned urls?

Servers often have limits on how much data can you upload at a time. Google Cloud run has a limit of 32MB, but if you use Compute Engine, it has no limits. Similarly AWS EC2 and ECS does not have limits but AWS Lambda has a limit of 10MB. So mostly serverless platforms have a limit not defined by you.

But in any case, uploading through backend won't be a good choice because you are unnecessarily eating up the bandwidth while uploading the file. This works for 10-20 files but if you have 1000 files of 100MB each, you can imagine what would happen.

How do we use presigned urls?

We request our server, that we want to upload a file, we provide the type of file and where we want to upload this file. The server should then request AWS S3 / GCS (or whatever service you are using) to create a presigned url. We can set the validity of this url.

Then we send this URL to the client. The client can then securely send the file on this URL.

There are practical limits for file size for presigned urls (5GB per file) but a good practice is, let's say if you have a file of 1GB, you break it into chunks of 10MB. You will have 100 chunks. Now you request for a presigned url for each chunk and then upload that chunk.

Once uploaded, you can combine all chunks on the S3 to make the file again. This is a common practice and this enables you to pause and resume your uploads as well.

AWS S3 allows 10,000 chunks per file and each chunk should be of size more than 5MB. These are some restrictions that AWS applies. This way, you can upload files of up to 5TB on AWS.

func (m *S3Service) getPresignedUrl(bucket constant.Bucket, key string) (string, error) {
    presignClient := s3.NewPresignClient(m.s3Client)
    url, err := presignClient.PresignPutObject(context.TODO(), &s3.PutObjectInput{
        Bucket: aws.String(string(bucket)),
        Key:    aws.String(key),
    }, s3.WithPresignExpires(15*time.Minute))
    if err != nil {
        return "", err
    }
    return url.URL, nil
}

The above is a go snippet to generate a presigned url. A presigned url might look like below

https://some-bucket.s3.region.amazonaws.com/some-prefix/some-key?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=some-credentials&X-Amz-Date=20250523T101848Z&X-Amz-Expires=900
&X-Amz-SignedHeaders=host&x-id=PutObject&X-Amz-Signature=some-signature

There are some headers like

1. X-Amz-Algorithm

2. X-Amz-Credential

3. X-Amz-Date

4. X-Amz-Expires

5. X-Amz-SignedHeaders

6. x-id

7. X-Amz-Signature

Now, using this, anyone can make a PUT request and upload a file to S3.

For multipart uploads, we will talk this in some other post.

Thanks.