AWS EC2 Instance Storage Options

Link to exam: https://aws.amazon.com/certification/certified-cloud-practitioner/

In our previous blog post, we talked about Amazon EC2, or Elastic Compute Cloud, which is a key service offered by Amazon Web Services (AWS). Simply put, EC2 gives you virtual servers in the AWS Cloud.

You can check out more about EC2 (Elastic Compute Cloud) at this link: AWS EC2 - Elastic Compute Cloud.

Now, let's shift our focus to another important part of EC2: storage.

EC2 Instance Storage, also called ephemeral storage, is a crucial part of EC2 instances that affects how well they work. In this blog post, we'll dive into EC2 Instance Storage, looking at its different types, what makes it unique, best ways to use it, and how it fits into different situations.

1. Ephemeral Storage: EC2 Instance Store Overview

• Definition: Ephemeral storage, also known as instance store volumes, is directly attached to EC2 instances.

• Proximity: Offers unparalleled performance due to direct attachment to underlying physical hardware.

• Performance: High I/O performance and low latency due to direct connection.

• Volatility: Data stored on instance store volumes is volatile and lost permanently upon instance termination or stoppage.

• Non-Persistence: Data is not retained if the instance is stopped or terminated.

• Responsibility: Backups and replication are the user's responsibility.

• Use Cases:

  • Good for buffer, cache, scratch data, or temporary content.

  • Not suitable for data requiring persistence or durability.

---

2. Elastic Block Store (EBS) Volumes Overview

EBS Features

• Network Drive: EBS volumes are not physical drives but rather network-attached storage. They operate over the network, allowing EC2 instances to access storage remotely.

• Persistence: EBS volumes stick around even if the EC2 instance is stopped or terminated. This makes them great for data that needs to last beyond the life of a single instance.

• Latency Impact: Communication between EBS volumes and EC2 instances occurs over the network, which may introduce some delay. For example, accessing data on an EBS volume might take slightly longer compared to accessing data stored locally on the instance.

• Flexibility: You can detach EBS volumes from one instance and attach them to another, which is handy for moving data around or switching between instances.

Types: There are different types of EBS volumes tailored for different needs, like General Purpose SSD, Provisioned IOPS SSD, and Throughput Optimized HDD.

Attachment Specifics:

• EC2 Instance Specific: Each EBS volume is linked to just one EC2 instance at a time, providing dedicated storage for that instance.

• Availability Zone Specific: EBS volumes are tied to specific Availability Zones, ensuring data resilience and optimizing data access within a region. So, keep in mind that an EBS volume created in one region won't work with an instance in another region.

Delete on Terminate Flag: This is an important setting for EBS volumes. When you create an EBS volume and attach it to an EC2 instance, you have the option to set the "Delete on Terminate" flag.

• If this flag is set to true, the volume will be deleted automatically when the EC2 instance is terminated.

• By default, this flag is set to true for the root EBS volume of an EC2 instance, meaning that when you terminate the instance, its root volume will be deleted automatically to avoid unnecessary storage costs.

EBS Snapshots: Smart Backups and Easy Mobility

• Efficient Backups: EBS snapshots capture the complete state of EBS volumes at a moment in time, making backups quick and efficient. They only store changes made since the last snapshot, which saves on storage costs and speeds up the process.

• Moving Data: With EBS snapshots, you can easily transfer data between Availability Zones. Simply create a snapshot of the source volume and copy it to the desired zone. This helps with disaster recovery and balancing workloads.

• Recovery after Failure: Should a system failure occur or data is lost, EBS snapshots come to the rescue. By restoring an EBS volume from a recent snapshot, you can swiftly recover lost data and resume normal operations, minimizing downtime and disruptions.

Benefits:

• Data Protection: Guards against data loss and corruption.

• Cost Savings: Saves money by only storing changed data.

• Flexibility: Moves data seamlessly across AWS infrastructure.

• Reliable Recovery: Gets you back up and running quickly after issues.

Best Practices:

• Regular Backups: Set up a consistent snapshot schedule.

• Lifecycle Management: Automate snapshot creation and retention.

• Testing: Regularly practice restoring backups to ensure they work.

• Disaster Planning: Use EBS snapshots in your disaster recovery plans for fast and reliable recovery.

Step-By-Step Hands-On Guide

To dive deeper into creating and managing EBS volumes, check out our step-by-step guide: AWS EBS Hands-On Guide.

---

3. Amazon Machine Images (AMIs) Overview

Definition and Purpose

• AMIs serve as templates for EC2 instances, containing preconfigured operating systems, applications, and settings.

• They streamline the deployment process by enabling users to launch EC2 instances with predefined configurations quickly.

• Region Specificity: AMIs are built for a specific AWS region, meaning they are only available for launching instances within that region's infrastructure. However, AMIs can be copied across regions, allowing users to replicate their custom images in different AWS regions for redundancy or scalability purposes.

Key Components

• Operating System: Base OS like Amazon Linux, Ubuntu, or Windows Server.

• Software Stack: Additional software packages, middleware, or applications required for specific workloads.

• Configuration Settings: Predefined security, networking, permissions.

Types of AMIs:

1. Public AMI: Provided by AWS, these are available to all users and commonly include popular operating systems and software configurations.

2. Custom AMI: Users can create and manage their own AMIs to meet specific needs, allowing for custom software configurations, security settings, and application stacks.

3. AWS Marketplace AMI: Created by third-party vendors, these AMIs are available for purchase or subscription through the AWS Marketplace and may offer specialized software solutions or preconfigured environments.

Customization Options

• Adding Software: Install additional packages or applications.

• Configuration Changes: Modify settings like firewall rules, user permissions, or system configurations.

• Snapshotting Instances: Existing EC2 instances can be used as a basis for creating custom AMIs, preserving the state of the instance for future deployments.

Lifecycle Management:

• Versioning: Support for maintaining multiple versions and track changes.

• Sharing/Distribution: Share with other AWS accounts or publicly distribute or sell them through the AWS Marketplace.

• Deregistration/Deletion: Remove unused or outdated AMIs to optimize costs

AMI Process

1. Creation: Users customize an EC2 instance and create an AMI from it.

2. Registration: The AMI is registered within a region with metadata.

3. Distribution:

   • Users choose sharing options: public, private, or across regions.

   • Public AMIs are accessible to all users, while private ones require explicit permissions.

   • AMIs can be copied across regions for redundancy or scalability.

4. Deployment:

   • Users select the AMI to launch instances with predefined settings.

   • AWS provisions a new EC2 instance based on the selected AMI, replicating the state captured during creation.

   • The instance is then ready for use with the predefined configurations and settings.

Use Cases:

• Application Deployment: Deploy web servers, databases, etc.

• Development/Testing: Ensure consistency across software development lifecycle.

• High Availability/Disaster Recovery: Enabling the rapid deployment of redundant instances in different regions or Availability Zones.

Automation via EC2 Image Builder:

• Automates the process of creating, testing, and distributing AMIs, streamlining the image management workflow.

• Allows users to define custom build pipelines with predefined configuration templates, ensuring consistency and repeatability in image creation.

• Supports integration with other AWS services and third-party tools, facilitating seamless image distribution across multiple regions and accounts.

• EC2 Image Builder is free, but users pay for EC2 instances, storage, and data transfer used during image creation. No extra charges for using Image Builder, just standard AWS fees for resources used.

Benefits:

• Efficiency: Rapid instance deployment, reduced setup time.

• Consistency: Uniform environment, minimized configuration drift.

• Automation: Scalable and reliable image management.

• Customization: Tailor instances to unique workload requirements.

Best Practices:

• Version Control: Track changes for reproducibility.

• Security Hardening: Implement best practices for protection.

• Regular Updates: Keep AMIs and configs up-to-date.

• Testing: Thorough testing before deployment

Step-By-Step Hands-On Guide

To dive deeper into creating and managing AMI, check out our step-by-step guide: AWS AMI Hands-On Guide.

---

4. EFS (Elastic File System) Overview

• Definition: EFS is a fully managed Network File System (NFS) provided by AWS, enabling shared access to files among multiple EC2 instances.

• Shared File Storage: Enables concurrent access by multiple EC2 instances to shared file storage.

• Managed NFS: Allows mounting on hundreds of EC2 instances, providing highly available and scalable file storage.

• Scalability: Automatically scales storage capacity based on usage, accommodating workload growth seamlessly, eliminating the need for capacity planning.

• Compatibility: Specifically designed for Linux and Unix-based instances, supporting NFSv4 protocol.

• Durability: Data is replicated across multiple Availability Zones within a region, ensuring high durability.

• Cost: Relatively expensive compared to EBS (3x gp2), but users only pay for the storage they consume, with no upfront commitments.

EFS vs. EBS:

EFS-IA (Elastic File System - Infrequent Access) Overview

• Definition: EFS-IA is a storage class within EFS tailored for storing infrequently accessed data.

• Cost Optimization: Specifically designed for files not accessed daily, offering significant cost savings compared to standard EFS.

  • Up to 92% lower cost compared to EFS Standard.

• Lifecycle Policy: Enable EFS-IA by implementing a Lifecycle Policy, automatically moving files to EFS-IA based on their last access time.

• Access Latency: Retrieval of data from EFS-IA may experience slightly higher latency compared to standard EFS.

  • However, transparent to applications accessing EFS, maintaining seamless operation.

• Use Cases: Ideal for storing data with infrequent access patterns, allowing users to optimize costs while maintaining accessibility.

---

5. Amazon FSx Overview:

Amazon FSx provides tailored managed file storage solutions for specific operating systems and use cases, offering high performance, scalability, and compatibility with existing environments.

Users can also launch third-party high-performance file systems on AWS through FSx, benefiting from fully managed services without infrastructure management complexity.

FSx for Windows:

• Definition: Network File System (NFS) for Windows servers.

• Functionality: Provides fully managed Windows file storage compatible with Windows Server Active Directory.

• Features:

  • Seamless integration with existing Windows environments.

  • Supports SMB (Server Message Block) protocol for file access.

  • Offers features like NTFS permissions, file locking, and deduplication.

• Use Cases: Ideal for Windows-based applications, databases, and shared file storage in Windows environments.

• Accessibility: Can be accessed from AWS instances or on-premise infrastructure.

FSx for Lustre (Linux + Cluster):

• Definition: High-Performance Computing (HPC) Linux file system.

• Functionality: Designed for high-performance, parallel computing workloads.

• Features:

  • Delivers sub-millisecond latencies and throughput of up to hundreds of GB/s.

  • Optimized for data-intensive workloads such as machine learning, genomics, and financial simulations.

  • Provides POSIX-compliant file system for compatibility with Linux-based HPC applications.

• Use Cases: Suitable for HPC applications requiring high throughput and low-latency file storage, particularly in fields like scientific research and engineering simulations, Machine Learning, Analytics, Video Processing, Financial Modeling

---

Summary

1. Ephemeral Storage (EC2 Instance Store):

   • High-performance, volatile storage directly attached to EC2 instances.

   • Suitable for temporary data like buffers or caches.

2. Elastic Block Store (EBS) Volumes:

   • Network-attached storage with persistence.

   • Supports various types and features like snapshots for backups.

3. Amazon Machine Images (AMIs):

   • Templates for EC2 instances with predefined configurations.

   • Supports versioning, sharing, and customization.

4. Amazon EFS (Elastic File System):

   • Fully managed NFS for shared file storage.

   • Scalable, compatible, and durable.

5. Amazon FSx:

   • Managed file storage for specific OS use cases, offer high performance, scalable, compatible.

   • FSx for Windows: Fully managed Windows file storage, ideal for Windows-based apps.

   • FSx for Lustre: High-performance Linux file system for HPC workloads.