Understanding AWS VPC Endpoints: Private Access to S3 Explained

Introduction

• When accessing Amazon S3 from EC2 instances, the default path routes through the public internet, creating potential security vulnerabilities. This guide demonstrates how to establish secure, private connections between your VPC and S3 using VPC endpoints.

• Implementing this networking feature will enhance security by keeping traffic within AWS's network while gaining more granular access control.

• Follow along as we learn by definitions of key terminologies and build this secure architecture step by step.

Introducing... VPC endpoints!

VPC endpoints give your VPC private, direct access to other AWS services like S3, so traffic doesn't need to go through the internet.

Just like how internet gateways are like your VPC's door to the internet, you can think of VPC endpoints as private doors to specific AWS services.

What we’ll be doing:

1. Use VPC endpoints to directly access your S3 bucket from your VPC.

2. Test your endpoint setup using an S3 security tool called bucket policies!

Architecture Diagram

Step 1: Set up Architecture

In this step, we're going to:

1. Create a VPC from scratch!

2. Launch an EC2 instance, which you'll connect to using EC2 Instance Connect later.

3. Set up an S3 bucket.

Creating a VPC

• Log in to your AWS Account.

• Head to your VPC console - search for VPC at the search bar at the top of your page.

• From the left-hand navigation bar, select Your VPCs.

• Select Create VPC.

• Select VPC and more.

• Under Name tag auto-generation, enter For-Endpoint

• The VPC's IPv4 CIDR block is already pre-filled to 10.0.0.0/16 - we'll use this default CIDR block!

• For the IPv6 CIDR block, we'll leave the default option of No IPv6 CIDR block.

• For Tenancy, we'll keep the selection of Default.

• For Number of Availability Zones (AZs), we'll use just 1 Availability Zone.

• Make sure the Number of public subnets chosen is 1.

• For Number of private subnets, we'll go with 0 private subnets.

• For the NAT gateways ($) option, make sure you've selected None. As the dollar sign suggests, NAT gateways cost money!

• For the VPC endpoints option, select None.

💡 What are VPC endpoints?

Normally, to access some AWS services like S3 from your VPC, your traffic would go out to the public internet.

But, VPC endpoints let you connect your VPC privately to AWS services without using the public internet. This means your data stays within the AWS network, which can improve security and reduce data transfer costs.

We're selecting None here, so we can learn to set one up manually later!

• You can leave the DNS options checked.

• Select Create VPC.

Launch an instance in your VPC

• Head to the EC2 console - search for EC2 in the search bar at the top of the screen.

• Select Instances at the left-hand navigation bar.

• Select Launch instances.

• Since your first EC2 instance will be launched in your first VPC, let's name it Instance-VPC-Endpoints

• For the Amazon Machine Image, select Amazon Linux 2023 AMI.

• For the Instance type, select t2.micro.

• For the Key pair (login) panel, select Proceed without a key pair (not recommended).

• At the Network settings panel, select Edit at the right-hand corner.

• Under VPC, select For-Endpoint-vpc.

• Under Subnet, select your VPC's public subnet.

• Keep the Auto-assign public IP setting to Enable.

• For the Firewall (security groups) setting, choose Create security group.

• Name your security group SG-VPC-Endpoints.

• That's it! No security group rules to add.

• Select Launch instance.

Launch a bucket in Amazon S3

• Before we connect with our EC2 instance, let's set up a bucket in Amazon S3.

• After creating this bucket, we'll access it from our EC2 instance and do things like checking what objects are in the bucket.

• Search for S3 at the search bar at the top of your console.

• Make sure you're in the same Region as your VPC!

• Select Create bucket.

• Let's set up a bucket! Keep the Bucket type as General purpose.

• Your bucket name should be vpc-endpoints-yourname

  • Don't forget to replace yourname with your first name! S3 bucket names need to be globally unique.

• We'll leave all other default settings, and go straight to Create bucket.

• We'll finish setup by uploading two files into our new bucket.

• This can be any two files on your local computer!

• Click into the bucket.

• Select Upload.

• Select Add files in the Files and folders panel.

• Select two files on your local computer to upload.

• Your files should show up in your Files and Folders panel once they're uploaded!

• Select Upload.

Step 2: Connect to EC2 Instance

• This tutorial is all about VPC endpoint connections, so before we create that endpoint... let's see what things are like without endpoint connections in place.

• In this step, we're gonna connect to your EC2 instance and try to access S3 through the public internet!

• Head to your EC2 console and the Instances page.

• Select the checkbox next to Instance-VPC-Endpoints.

• Select Connect.

• In the EC2 Instance Connect setup page, select Connect again.

• Successfully connected!

• For your first command, try running aws s3 ls, which is a command used to list the S3 buckets in your account (ls stands for list!).

• It doesn't look like a success. We need to provide credentials.

💡 What are credentials?
Credentials in AWS are like keys that let you access and manage AWS services securely. Without credentials, you won't have the permission to do things like viewing your S3 bucket list.

💡 Do I have credentials?
When you log into your AWS Management Console, you're already authenticated using your AWS user's details.

Running commands from an EC2 instance is a different story! You can think of your EC2 instance as another user that needs its own username and password (i.e. credentials) to get access to your AWS account.

Your account by default doesn't have credentials set up for running commands from an EC2 instance. You'll need to manually set these up to securely access and manage AWS services from your instance.

• Run aws configure

• The terminal is now asking us for an Access Key ID!

💡 What is an access key ID? Do I have one?
An access key ID is a part of a credential!

Your credentials are made up of a username and password; think of the access key ID as the username.

You don't automatically have one, but you can create access keys IDs through AWS IAM.

💡 What's the difference between access keys and key pairs?
You might remember key pairs from launching EC2 instances!

• Key pairs are used specifically for logging into your EC2 instances through SSH.

• Access keys, which are what we're learning about now, are credentials for your applications and other servers to log into AWS and talk to your AWS services/resources.

Step 3: Create Access Keys

In this step, we're going to:

• Give our EC2 instance access to your AWS environment.

• Search IAM console that helps us set this up. Select IAM.

• Search for Access keys again in the IAM console's left-hand navigation panel.

• Choose the search result for managing an access key for your IAM Admin account.

• On the first setup page, select Command Line Interface (CLI).

• Select the checkbox that says I understand the above recommendation and want to proceed to create an access key.

• Select Create access key.

💡 What is the yellow warning banner saying?
In this tutorial we're creating access keys and manually applying them in our EC2 instance, but typically the recommended way is to create an IAM role with the necessary permissions and then attaching that role to your EC2 instance.

Your EC2 instance would inherit the permissions from the role, and this is best practise as you can easily attach and detach EC2 instances from roles to give and take away their credentials.

We aren't using this method so that we can learn about access keys, but roles are usually a better alternative for security.

• This is the only time that your secret access key can be viewed or downloaded. You cannot recover it later. However, you can create a new access key any time.

• Select Download .csv file near the bottom of the page.

💡 What is the secret access key?
The secret access key is like the password that pairs with your access key ID (your username). You need both to access AWS services.

Secret is a key word here - anyone who has it can access your AWS account, so we need to keep this away from anyone else!

Step 4: Connect to your S3 bucket

In this step, we're going to:

1. Head back to your EC2 instance.

2. Get your EC2 instance to access your S3 bucket.

• Now back to your EC2 Instance Connect tab!

• Open the AccessKeys.csv file you've downloaded.

• Copy the Access key ID.

• Paste this into your EC2 instance's terminal, and press Enter on your keyboard.

• Let's do the same for the AWS Secret Access Key!

• Copy the key from your .csv file, and paste it in the terminal. Press Enter.

• Let's try running the aws s3 ls command again to see a list of your account's S3 buckets.

• Do you see a list of your buckets? You might have a shorter list than the one in this screenshot, but the only line to look out for is vpc-endpoints-yourname.

• Next, let's run the command aws s3 ls s3://vpc-endpoints-yourname. Make sure to replace ‘vpc-endpoints-yourname’ with your actual bucket name!

• We can see the objects that are inside your bucket.

• Run sudo touch /tmp/example.txt to create a blank .txt file in your EC2 instance.

💡 What does this command say?
Let's break it down!

• sudo stands for "superuser do" and it's used when you're running a command that typicaly needs elevated user permissions, like installing software or creating new files.

• touch is a standard command used to create an empty file if it doesn't exist. If it does exist, this command will update the file's timestamp (i.e. the last time it was accessed) instead.

• /tmp/ is a common directory path (you can think of a directory path as layers and layers of folders) typically used for temporary files.

• example.txt is the name of the empty file you're creating!

• Next, run aws s3 cp /tmp/example.txt s3://vpc-endpoints-yourname to upload that file into your bucket.

💡 What does this command say?
Let's break it down!

• aws s3 cp is the command to copy files.

• /tmp/example.txt is the source file path. It's saying that the file to copy is example.txt, which exists inside a folder called tmp.

• s3://vpc-endpoints-yourname is the destination path. It's saying that the file should be copied to the S3 bucket vpc-endpoints-yourname!

• Switch tabs back to the S3 console.

• Refresh the Objects tab for your S3 bucket.

• The file is copied into the S3 bucket!

Step 5: Create an endpoint

• Your EC2 instance definitely has access to your bucket - access keys worked!

• But your instance is connected to your bucket through the public internet.

• This isn't the most secure way to communicate - external threats and attacks can easily intercept your commands and get access to your AWS environment or sensitive data.

In this step, we're going to:

1. Set up a way for the VPC and S3 to communicate directly.

• In a new tab, head back to your VPC console.

• Select Endpoints from the left-hand navigation panel.

• Select Create endpoint.

💡 What's an endpoint?

An endpoint in AWS is a service that allows private connections between your VPC and other AWS services without needing the traffic to go over the internet.

💡 I thought all of my resources get launched inside my VPC, so how are some connections through the internet?

Not all AWS resources get launched inside your VPC! While your EC2 instances live within your VPC, S3 buckets and some other AWS services exist outside your VPC because they're designed to be highly available and accessible from anywhere.

For example, the commands you're putting through AWS CLI now will be communicated to your S3 bucket through the public internet.

That's why VPC endpoints exist to create a private connection between your VPC and AWS services. Having a VPC endpoint means your instances can now access services like S3 directly without routing through the public internet, which makes sure your data stays within the AWS network for security.

• For the Name tag, let's use VPC-Endpoint-S3.

• Keep the Service category as AWS services.

• In the Services panel, search for S3.

• Select the filter result that just ends with s3.

• Select the row with the Type set to Gateway.

💡 What's does Gateway mean?
A Gateway is a type of endpoint used specifically for Amazon S3 and DynamoDB (DynamoDB is an AWS database service).

Gateways work by simply adding a route to your VPC route table that directs traffic bound for S3 or DynamoDB to head straight for the Gateway instead of the internet.

• Next, at the VPC panel, select For-Endpoint-vpc.

• We'll leave the remaining default values and select Create endpoint.

• Endpoint is created!

Step 6: Create a super secure bucket policy

• We've just set up a VPC endpoint.

• You might be wondering:
  - How do I know that my endpoint is truly providing direct access to S3?
  - How can I confirm that my VPC isn't still communicating with S3 through the public internet?

• To answer these questions, we'll validate our endpoint connection by blocking off your S3 bucket from ALL traffic... except traffic coming from the endpoint.

• By testing this, we’ll know if the VPC endpoint was truly set up properly; then your EC2 instance should still be able to access S3. Otherwise, your instance's access is blocked!

In this step, we're going to:

1. Limit our S3 bucket access to only traffic from our endpoint.

In our S3 bucket, we're going to kick things up a notch and make access super secure.

Let's say we want to make this bucket completely private to ALL access... except access through the VPC endpoint you've set up.

• In the S3 console, select Buckets from the left-hand navigation panel.

• Click into your bucket vpc-endpoints-yourname.

• Select the Permissions tab.

• Scroll to the Bucket policy panel and select Edit.

💡 What's a bucket policy?
A bucket policy is a type of IAM policy designed for setting access permissions to an S3 bucket. Using bucket policies, you get to decide who can access the bucket and what actions they can perform with it.

• Add the below bucket policy to the policy editor.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpce": "vpce-xxxxxxx"
        }
      }
    }
  ]
}

💡 What does this policy do?
This policy denies all actions (s3:*) on your S3 bucket and its objects to everyone (Principal: "*")... unless the access is from the VPC endpoint with the ID defined in aws:sourceVpce.

In other words, only traffic coming from your VPC endpoint can get any access to your S3 bucket!

• Don't forget to replace:

  1. BOTH instances of arn:aws:s3:::your-bucket-name with your actual bucket ARN.

  2. vpce-xxxxxxx with your VPC endpoint's ID.

• To find this ID, you'll have to switch back to your Endpoints tab and copy your endpoint's ID. Make sure it starts with vpce-

• Select Save changes.

• Once the changes are saved, the panels turn red all across the screen.

💡 Why am I denied access?!
Don't forget what your policy does,

Your policy denies all actions unless they come from your VPC endpoint. This means any attempt to access your bucket from other sources, including the AWS Management Console, is blocked!

Step 7: Access your S3 bucket again!

• Now that S3 bucket's access is only limited to our endpoint, can our EC2 instance still access your bucket?

• If it can, we've set up our endpoint connection perfectly.

• If it can't, something's gone wrong with our VPC endpoint setup.

In this step, we're going to:

1. Test our VPC endpoint set up.

2. Troubleshoot a connectivity issue.

• Head back to EC2 Instance Connect.

• Try running aws s3 ls s3://vpc-endpoints-yourname again.

• Access denied! :( It looks like the bucket policy had stopped us.

💡 Why is our EC2 instance also denied access?
We're supposed to get exclusive access to this S3 bucket... after all, the policy denies everyone except traffic through the endpoint!

We did set up an endpoint already between your VPC and your S3, didn't we?

• Troubleshoot this error by heading to your VPC console.

• Select Subnets from the left-hand navigation panel.

• Select your public subnet, which starts with NextWork-subnet-public1.

• Select the Route table tab.

• The issue is solved! :)

💡 How is it Issue solved?!
Another question to answer first - using this route table, how would traffic from your EC2 instance get access to your S3 bucket?

The route table is the GPS / traffic controller directing traffic from your EC2 instances.

If your route table doesn't have a route that directs traffic bound for S3 to your VPC endpoint, traffic from your EC2 instance is actually trying to get to your S3 bucket through the public internet instead.

• Let's make sure your public subnet has a route to your endpoint.

• Select Endpoints from the left hand navigation panel.

• Select the checkbox next to your endpoint, and select Route tables.

• Select Manage route tables.

• Select the checkbox next to your public route table.

• Select Modify route tables.

• Head back to the Subnets page.

• Select the refresh button next to the Actions dropdown.

• Check the Route table tab for your public subnet again.

• Good! there's a route to the VPC endpoint now.

Step 8: Access your S3 bucket one more time!

• Our VPC endpoint setup is perfect now!

• To validate your work, let's get your EC2 instance to interact with your S3 bucket one last time.

In this step, we're going to:

1. Test our VPC endpoint set up (again).

2. Restrict our VPC's access to our AWS environment.

• Back in your EC2 Instance Connect tab, run aws s3 ls s3://vpc-endpoints-yourname again.

• Finally, a successful VPC endpoint connection! :)

Did you know that VPC endpoints can be configured using policies too?!

• We'll do a simple configuration - switch to your Endpoints tab.

• Select the checkbox next to your policy.

• Select the Policy tab.

• Ooo! By default, your VPC endpoint allows access to all AWS services.

• Select Edit policy.

• Change the line "Effect": "Allow" to "Effect": "Deny"!

• Click Save.

• Switch back to your EC2 Instance Connect tab.

• Run aws s3 ls s3://vpc-endpoints-yourname - What’s the output?

• The endpoint policy now stops us from accessing our S3 bucket.

💡 Why would I use policies to configure a VPC endpoint?
This is a great tool to quickly block off access if you suspect attackers are using your endpoint to access your resources! Instantly switch the effect to Deny, and your Gateway is closed.

You could also use VPC endpoint policies to be even more granular with the way you give away access to your AWS services. For example, you can write a policy that gives your endpoint read access only to S3 buckets, so the permission to upload objects is denied.

• Switch tabs back to your Endpoints page.

• Select Edit Policy.

• Change the policy back to "Effect": "Allow" This will be important when it comes to deleting our resources later!

• Select Save changes.

Great Work!

We've just completed today's tutorial and learned how to set up direct, private access to S3 from your VPC.

Deletion of Resources

Delete S3 Buckets

• Head to your S3 console.

• Select the Buckets page.

• Select your nextwork-vpc-endpoints-yourname, and select Delete.

• Enter your bucket name and select Delete bucket.

• We can not delete it!

• Deleting your bucket from the console is not possible - don't forget that your S3 bucket policy literally denies everyone else (except traffic from the VPC endpoint) access to the bucket.

• Our only option is to delete our bucket through the EC2 instance!

• Switch tabs to your instance's terminal.

• Run the command aws s3 rb s3://vpc-endpoints-yourname

💡 What does this command do?
rb is a command that deletes your bucket!

This means rb s3://vpc-endpoints-yourname is used to delete your S3 bucket vpc-endpoints-yourname.

• The delete failed - we need to make sure our bucket is empty first.

• To delete everything in your bucket, run the command aws s3 rm s3://vpc-endpoints-yourname --recursive

💡 What does this command do?

• rm stands for "remove" and is used to remove things inside your bucket.

• --recursive means the remove command should include all objects and subdirectories inside your bucket. This is a super helpful command to run before delete a non-empty bucket.

• Run the command aws s3 rb s3://vpc-endpoints-yourname again.

• Let's check that it's gone by running aws s3 ls one last time.

• That's your S3 bucket successfully removed.

• Delete/Terminate your EC2 Instance.

• Delete your VPC Endpoint.

• Delete your VPCs.

• Delete Your IAM Access Keys.

Conclusion

• VPC endpoints provide an elegant solution for securing connections between your VPC and Amazon S3. By implementing this approach, we've eliminated public internet exposure while gaining precise control through bucket and endpoint policies.

• This same pattern can be applied to many other AWS services, making it a fundamental best practice for production environments. As we've seen, with just a few configuration steps, we can significantly enhance your AWS infrastructure's security posture.

  Happy Learning!! :)

  Connect With Me: LinkedIn | GitHub