Day 89 of 90 Days of DevOps Challenge: Mounting an AWS S3 Bucket on Amazon EC2 Linux Using S3FS

Amazon S3 (Simple Storage Service) is a popular object storage service that provides developers and IT teams with secure, durable, and highly-scalable storage for their data. In this blog post, we will walk through the steps to mount an AWS S3 bucket on an Amazon EC2 Linux instance using S3FS, which allows you to treat your S3 bucket as a local filesystem.

---

Project Overview

This project will provide a hands-on experience with key AWS services, including EC2 and S3, while also utilizing S3FS for seamless integration. By the end of this project, you will have mounted an S3 bucket on your EC2 instance and learned how to interact with the bucket using standard file commands.

---

Key Components

1. AWS S3

Amazon S3 is an object storage service designed to store and retrieve any amount of data from anywhere on the web. It offers high availability, security, and performance.

2. Amazon EC2

Amazon EC2 (Elastic Compute Cloud) is a web service that provides resizable compute capacity in the cloud, allowing you to run applications and services in a flexible and scalable environment.

3. S3FS

S3FS is a FUSE (Filesystem in Userspace) based filesystem that allows you to mount an S3 bucket as a local filesystem on Linux. This provides easy access to S3 bucket files using standard file commands.

---

Implementation Steps

Step 1: Create IAM User and Set Policies

1. Create IAM User: Log in to your AWS Management Console and navigate to the IAM service to create a new user. Assign permissions that allow access to S3, such as the AmazonS3FullAccess policy or a custom policy that meets your project requirements.

2. Generate Access Keys: During user creation, ensure you generate and save the access key ID and secret access key, as you will need these for authentication with the AWS CLI.

Step 2: Install AWS CLI

1. Connect to Your EC2 Instance: Launch an EC2 instance (preferably a Linux instance), and SSH into it:

    ssh -i <your-key.pem> ec2-user@<your-ec2-instance-ip>
   

2. Install AWS CLI: Install the AWS CLI on your EC2 instance if it's not already installed:

    sudo yum install -y aws-cli
   

3. Configure AWS CLI: Run the following command to configure the AWS CLI with your IAM user credentials:

    aws configure
   

   Enter your Access Key ID, Secret Access Key, default region, and output format when prompted.

Step 3: Install S3FS

1. Install Dependencies: Before installing S3FS, ensure you have the necessary packages:

    sudo yum install -y gcc-c++ make
    sudo yum install -y fuse
   

2. Install S3FS: You can install S3FS from the EPEL repository or compile it from source. To install it from the EPEL repository:

    sudo amazon-linux-extras install epel -y
    sudo yum install -y s3fs-fuse
   

Step 4: Mount the S3 Bucket

1. Create a Directory: Create a directory where you want to mount the S3 bucket:

    mkdir ~/s3bucket
   

2. Create a Password File: Create a password file to store your AWS credentials for S3FS. Make sure to restrict permissions to this file:

    echo "<your-access-key-id>:<your-secret-access-key>" > ~/.passwd-s3fs
    chmod 600 ~/.passwd-s3fs
   

3. Mount the S3 Bucket: Use the following command to mount your S3 bucket:

    s3fs <your-bucket-name> ~/s3bucket -o passwd_file=~/.passwd-s3fs -o allow_other
   

   Replace <your-bucket-name> with the name of your S3 bucket.

4. Verify the Mount: You can verify if the bucket is mounted by listing the contents:

    ls ~/s3bucket
   

Step 5: Run the Project and Share

Now that you have successfully mounted the S3 bucket on your EC2 instance, you can interact with it like any other directory on your system. You can upload, download, and manage files using standard file commands such as cp, mv, and rm.

1. Upload a File: For example, to upload a file to your S3 bucket:

    echo "Hello, S3!" > ~/s3bucket/hello.txt
   

2. Check on S3: You can log in to your AWS Management Console, navigate to S3, and check if the hello.txt file appears in your bucket.

3. Share on LinkedIn: After completing the project, don’t forget to share your success on LinkedIn and showcase your newfound skills with AWS and S3FS!

---

Conclusion

By following the steps outlined in this blog post, you can effectively mount an AWS S3 bucket on an Amazon EC2 Linux instance using S3FS. This project illustrates the power of AWS services and how they can be integrated to create scalable and efficient storage solutions.