Secure Way to Share S3 Objects: Using Pre-Signed URLs with Python (Boto3)

Amazon S3 is a highly scalable and reliable storage service, but when it comes to sharing your stored objects securely, simply making them public isn't always the best approach. Fortunately, AWS offers secure methods for sharing objects from private buckets, such as pre-signed URLs.

In this blog, we'll cover different ways to share S3 objects securely, with a focus on generating pre-signed URLs for temporary access using Python and Boto3.

What Are Pre-Signed URLs?

A pre-signed URL is a URL that provides temporary, secure access to an S3 object. You can generate a pre-signed URL using Boto3, allowing users to download or upload objects without needing AWS credentials. These URLs are time-limited, so once they expire, the access is no longer valid.

Why Use Pre-Signed URLs?

• Security: Users don’t need AWS credentials to access files.

• Control: You set an expiration time for the URL, giving you complete control over how long the resource is accessible.

• Granularity: You can specify actions (e.g., GET or PUT) that the URL will allow.

How to Share S3 Objects Securely:

Pre-signed URLs are the go-to solution for securely sharing objects. They provide temporary access to the object without making the entire bucket public.

Use Case 1: Generating Pre-Signed URL for Downloading

You can allow users to securely download a file using a GET request via a pre-signed URL.

import boto3
from botocore.exceptions import ClientError

# Initialize the S3 client
s3_client = boto3.client('s3')

def create_presigned_url(bucket_name, object_name, expiration=3600):
    """Generate a pre-signed URL to share an S3 object for download."""
    try:
        response = s3_client.generate_presigned_url('get_object',
                                                    Params={'Bucket': bucket_name, 'Key': object_name},
                                                    ExpiresIn=expiration)
    except ClientError as e:
        print(f"Error generating pre-signed URL: {e}")
        return None

    return response

# Example usage
url = create_presigned_url('your-bucket-name', 'your-file.txt')
if url:
    print(f"Pre-signed URL for download: {url}")

---

2. Pre-Signed URLs for Uploading

Pre-signed URLs are also useful when you want to allow a third party to upload a file to your S3 bucket securely, without sharing your credentials.

Use Case 2: Generating Pre-Signed URL for Uploading

To upload files/objects to S3, we have two options presigned POST and presigned URL with PUT. Both presigned POST and presigned URL with PUT are methods to allow users to upload files to an Amazon S3 bucket, but they are used in different scenarios and have different behaviors.

Here’s a breakdown of the key differences between presigned POST and presigned URL with PUT:

1. HTTP Method:

• Presigned POST:

  • Uses the POST method for uploading objects to S3.

  • Uploads happen via an HTML form (multipart form-data).

• Presigned URL with PUT:

  • Uses the PUT method to upload objects.

  • The user can upload an object in a single step using an HTTP PUT request.

2. How They Are Used:

• Presigned POST:

  • Typically used in web applications where you want to allow file uploads using an HTML form. This allows the user to include additional fields (like metadata or ACL) along with the file.

  • More flexible in handling file uploads with multipart/form-data.

  • Client-side applications (web browsers) can use this method to securely upload files directly to S3.

• Presigned URL with PUT:

  • Used when you need a direct file upload from the client (for example, an API or CLI).

  • The file content is sent directly in the body of the PUT request, without multipart form-data.

3. Content-Type and Metadata:

• Presigned POST:

  • You can specify conditions like the content type, object size, metadata, or ACL in the form fields.

  • The client needs to send these conditions in the form along with the file. These conditions are predefined when generating the presigned POST URL.

• Presigned URL with PUT:

  • Metadata like Content-Type and ACL can be specified during the URL generation, but they must be included as headers in the HTTP PUT request.

  • Only one object can be uploaded, with limited flexibility in setting metadata.

4. File Upload Flexibility:

• Presigned POST:

  • Supports multipart/form-data, which means you can include additional fields with the uploaded file, such as tags, ACLs, metadata, and more.

  • Offers greater control over conditions (e.g., file size, specific keys, etc.).

• Presigned URL with PUT:

  • Uploads the file directly in the request body.

  • Simpler but less flexible. Limited to just uploading the file without including additional data (metadata, ACLs, etc., must be set through headers).

5. Conditions and Validation:

• Presigned POST:

  • You can enforce various conditions on the upload, such as:

    • Maximum file size.

    • Content type.

    • Specific object key prefixes.

    • Bucket-level permissions.

  • These conditions are always validated by S3 before the object is accepted.

• Presigned URL with PUT:

  • Offers fewer validation options. The primary validation happens at the time of URL generation (e.g., object key, ACL, content-type headers).

  • There’s no built-in support for file size limitations or complex conditions in the URL itself.

6. Client-Side Complexity:

• Presigned POST:

  • Requires form fields to be set on the client side, making the implementation more complex for the client but provides better control over the file upload process.

  • Works well for browser-based file uploads where form-based submissions are common.

• Presigned URL with PUT:

  • Simpler on the client side, as the client just needs to make a PUT request to the URL with the file in the body.

  • Works well for non-browser clients like APIs, CLI tools, or direct HTTP clients.

7. Use Cases:

• Presigned POST:

  • Best for web applications where users upload files via a web form.

  • Useful when you need to add metadata, control file size, or define other constraints before uploading.

  • Common for browser-based applications and multi-part uploads.

• Presigned URL with PUT:

  • Best for API-based uploads, CLI tools, or when you want to upload a single file in one step.

  • Useful when you need a simple, direct upload without additional form fields or validations.

  • Common for server-to-server uploads or mobile app uploads.

8. Security:

• Presigned POST:

  • Offers more control through various conditions and constraints that S3 enforces before accepting the upload.

  • Generally better suited for public-facing applications where you want stricter control over what and how users upload files.

• Presigned URL with PUT:

  • The URL is secured by the signature and the expiration time, but we need to handle it properly, as it doesn’t have as many condition checks.

  • Typically better for controlled environments like internal systems or direct API uploads.

Comparison Table:

Example Code for Both:

1. generate_presigned_post Example:

    import boto3
   
    s3_client = boto3.client('s3')
   
    bucket_name = 'your-bucket-name'
    object_name = 'your-file.txt'
   
    # Generate a presigned POST URL
    response = s3_client.generate_presigned_post(
        Bucket=bucket_name,
        Key=object_name,
        Fields={
            'acl': 'public-read',
            'Content-Type': 'text/plain'
        },
        Conditions=[
            {'acl': 'public-read'},
            ['content-length-range', 1, 1048576],  # 1 byte to 1MB
            {'Content-Type': 'text/plain'}
        ],
        ExpiresIn=3600  # URL expires in 1 hour
    )
   
    print("Presigned POST data:", response)
   

2. generate_presigned_url with PUT Example:

    import boto3
   
    s3_client = boto3.client('s3')
   
    bucket_name = 'your-bucket-name'
    object_name = 'your-file.txt'
   
    # Generate a presigned URL for PUT
    presigned_url = s3_client.generate_presigned_url('put_object',
                                                     Params={'Bucket': bucket_name, 'Key': object_name},
                                                     ExpiresIn=3600)
   
    print(f"Presigned URL for PUT: {presigned_url}")
   

Explore the Working Code

You can explore the full code in my GitHub repository:

🔗 Explore the Working Code

Pre-signed URLs offer an ideal solution if you prefer programmatic solutions in Python over directly managing your AWS resources through the console. Whether you're integrating these URLs into an API, web application, or mobile app, their flexibility ensures security and ease of use.

Happy Learning 😊