Storage – EBS and S3

EBS

Volume Types

• majorly catagorized into SSD and HDD based volumes
• SSD -> high performance (i02, io1) for most demanding transactional apps and general purpose (gp2, gp3) for cost/performance balance
• HDD -> throughput optimized (st1) for freq. accessed workloads and cost/throught balanced low cost (sc1) for less freq. accessed data

Pricing

• pricing for volumes, clones, snapshots, recycle bin

Notes

• persistent block level storage
• can be backed up, resized and attached to different EC2 instances
• you should create backups (incremental) by creating snapshots because this is your permanent storage and if AZ hits disaster, without backup, you data might lost
• use as permanent storage of apps, db hosting, etc.
• data can be easily migrated between AZs using snapshots
• can be modified to different types/sizes a/c to usage patterns as you can switch between storage types and adjust capacity in zero downtime
• AWS keeps multiple copies of the data behind an EBS volume within the same AZ

EBS Snapshots

• EBS snapshots are point-in-time backups of EBS
• snapshots can be used for data migration, volume resizing, data backup and recovery
• snapshots are incremental so only blocks that have been changed after the recent snapshot are saved into next snapshot
• snapshots can also be used to create new volumes which will contain exact same data as OG volume at the time snapshot was taken
• snapshots are stored in multiple AZs using S3 for high availability and backups
• you can automate creation, retention and deletion of snapshots using AWS Data Lifecycle Manager
• AWS Data Lifecycle Manager schedules snapshots during off-peak hours which minimizes performance impact
• AWS Data Lifecycle Manager can also automatically delete outdated or old backups to save cost

Customer Responsibility

• customers are responsible for scheduling and managing snapshots for backups
• monitoring snapshot costs and deleting unnecessary snapshots to avoid extra charges
• make sure sensitive data within snapshots is encrypted

EFS

Pricing

• you pay for storage and data access, see more at EFS pricing by AWS

Notes

If NFS is hosted lecay way, you have to plan for scaling, storage, redundancy, security, etc. But with EFS, AWS does all that for you.

• AWS managed Network File System
• can be shared, read and write by multiple instances simultaneously
• comes with pre configured lifecycle policy which helps optimize costing
• you can also customize lifecycle a/c to your necessity

Compare with EBS

• EBS is AZ resource i.e. to attach EBS to EC2 both need to be in same AZ but EFS is not AZ limited, you can attach from the same region or across regions
• EBS is essentially a hard drive, you can fill it up with database files, logs etc. Once it’s full (say 2TB) it does not automatically scale but EFS scales automatically

S3

Elements

Objects

• any file uploaded to s3 becomes an object
• object includes data, metadata, uid or key
• uniquely identified by key which is generally its filename

Buckets

• container for storing objects
• has globally unique name

Metadata

• can be system-defined or user-defined
• user-defined are more for app usage where you can put conditions on the basis of those metadata

Storage Class

• ranges from standard to deep archive

Pricing

At least two type of pricing:

• storage pricing -> obviously you are storing your data
• requests pricing -> any PUT/COPY/POST/LIST, GET/SELECT requests you are charged

Each storage class has different pricing. One may be cheap for storage but expensive requests and vice-versa. So, when looking for price optimization, consider both cases to grab the better deal.

For more information, see s3 pricing and this blog.

Notes

• object store that can store any time of data with no upper limit
• suited for handling large unstructured data (images, docs, videos)
• everything stored in bucket is private by default but can be made public or access can be controlled
• bucket policy can be attached to bucket which determines which actions are allowed/denied on particular bucket alongwith objects inside that bucket
• identity-based-poilicies can be attached to users or identities which defines what actions users/groups/roles can perform on s3

AWS Storage Gateway

Types of AWS Storage Gateway

Common Use Cases and Benefits

In simple terms: AWS Storage Gateway helps organizations connect their existing data center with AWS cloud storage, making cloud adoption easier without requiring a complete infrastructure change.