TL;DR: Storage alone is not your bill. For a large image app (~500 TB, ~30 images per page), we lowered cost in two layers: move cold-but-hot data to S3 Glacier Instant Retrieval (GIR), then put CloudFront in front to tame egress and latency. Estimated monthly spend dropped from ~$24,600 (S3 Standard + direct access) to ~$15,700 (GIR + CloudFront with ~90% cache hit) — saving ~$8,900/month (~36%) without sacrificing instant image loads.

The Real Problem: Big Storage, Bigger Bills

We run an image-heavy application — the kind where storage is measured in hundreds of terabytes, not gigabytes. At our scale, we're sitting on roughly 500 TB of image data, and a typical page loads about 30 images at once.

On paper, that sounds like a storage problem. In practice, the pain shows up everywhere:

• Users expect instant images. Pages feel broken if thumbnails or full-resolution assets lag.

• Traffic is real and spiky. Mobile clients, multiple image sizes per asset, and repeat visits all multiply requests.

• The bill has layers. You don't pay once for "using 500 TB." You pay for storage class, API requests, retrieval, and — often the surprise — data transfer out (egress).

What looked like a simple "we have a lot of files" problem turned into a systems problem: how do we store cheaply, serve fast, and keep operations sane at scale?

A Mental Model for S3 Costs

Before changing architecture, it helps to see S3 as four stacked cost drivers (think of a stack — egress sits on top):

1. Storage ($/TB-month by class) — where most teams start (bottom of the stack)

2. Retrieval / access fees — depends on storage class

3. Requests (GET, PUT, LIST, etc.)

4. Data transfer out (egress) — often hurts most at scale (top of the stack)

Most optimization journeys begin at the bottom — storage class — and stop there. That works until traffic grows. Then egress climbs the stack and becomes the loudest line item.

For our workload — archive-like volume with immediate read expectations — that stack pointed us toward a two-step plan.

Step 1: Reduce the "Per-TB" Storage Bill (Standard → GIR)

Why not just keep everything in S3 Standard?

S3 Standard is great for frequently accessed, latency-sensitive data. But at 500 TB, the monthly storage line item alone is material — even before requests and egress.

We needed a class that says:

• Treat this like long-lived image archive data

• Still serve reads immediately (no multi-hour restore jobs)

That profile maps well to S3 Glacier Instant Retrieval (GIR).

What we expected for UX

From the application's perspective, objects still live behind our URLs/API. Clients should not feel a "cold storage" workflow. GIR is designed for data accessed roughly once per quarter but still retrievable in milliseconds — a reasonable fit for large image catalogs where many assets are rarely viewed but must remain instantly available when requested.

The honest trade-offs

Moving to GIR is not free optimization. You trade lower storage rates for different economics elsewhere:

• Storage rate — Lower than Standard (main win)

• Retrieval pricing — Per-GB fees apply on access

• Request pricing — Different from Standard

• Minimum storage duration — Objects deleted early can incur charges

So GIR is a win when most bytes sit quietly and access patterns are not hyper-hot across the entire 500 TB corpus. It is not a magic "make everything cheaper" button if every object is fetched constantly.

Rule of thumb we used: optimize where data is large, durable, and mostly idle — not where every byte is hit on every page load.

A Practical Note on "Many Folders" (Prefixes)

At our scale, folder structure is an operations decision, not a caching trick.

In S3, a "folder" is just a prefix. Examples:

• tenant/{tenantId}/yyyy/mm/{assetId}/original.jpg

• tenant/{tenantId}/yyyy/mm/{assetId}/thumb_320.jpg

• shard/{hashPrefix}/...

We use prefixes to:

• Separate tenants or domains for governance and access boundaries

• Partition by time (yyyy/mm) for lifecycle jobs and migrations

• Shard by hash to avoid one flat namespace that becomes painful to operate

What prefixes do not do: they do not replace a CDN or browser cache. Prefixes help humans and batch jobs — lifecycle policies, backfills, audits, and incident isolation. They make a 500 TB bucket manageable; they do not, by themselves, reduce egress.

The Twist: Storage Got Better — Egress Still Hurt

After shifting a large share of data to GIR, our storage bill improved. Then we looked at traffic.

Consider one page view:

• ~30 images per page

• Each image may exist in multiple variants (thumbnail, medium, full)

• Mobile clients, cache misses, and direct origin links all pull bytes from AWS

Even modest per-GB egress pricing adds up fast. Example math:

• 1,000,000 page views/day

• x 30 images/page

• x ~200 KB average per image (example)

• = ~1.8 TB transferred per day from origin — before retries, bots, or API-driven fetches

Storage optimization does not automatically fix this. In some cases, cheaper storage + same traffic can make egress feel more prominent in the monthly report.

That is the twist: the bottleneck moves up the stack.

Step 2: CloudFront + Cache — Fewer Origin Hits, Faster Pages

The second layer targets how bytes leave AWS, not just where they sit.

Request flow

1. User browser/app sends request

2. CloudFront edge (nearest POP) receives it

3. CACHE HIT — return image immediately (no S3 round trip)

4. CACHE MISS — fetch from S3 origin, cache at edge, return to user

For an image platform, this matters twice:

1. Cost: cached responses reduce S3 GET volume and origin egress

2. Experience: edge delivery cuts latency — especially valuable when each page loads ~30 assets

What actually moves the needle on cache hit ratio

Clever one-off tricks matter less than stable caching contracts:

• Set meaningful Cache-Control on objects (public, max-age aligned with how often images change)

• Use versioned URLs when assets update (.../v3/thumb.jpg or hash in path) so you can cache aggressively without stale content

• Tune TTLs per asset type — thumbnails and immutable originals can often cache longer than dynamic overlays

• Monitor hit ratio as a product metric, not only an infra metric

A page with 30 images is a parallel burst of requests. Good edge caching turns repeat views — and overlapping assets across pages — into mostly local edge responses.

Dual benefit

• GIR at origin — Cheaper resting storage for large, mostly idle catalog data

• CloudFront at edge — Cheaper/fewer origin transfers + lower latency for end users

Final Architecture: Two Layers, One Goal

Put together, the architecture is intentionally simple:

Clients (~30 imgs/page)
→ CloudFront (cache + edge)
→ S3 GIR (~500 TB, prefix layout) — on cache miss only

GIR answers: Where should half a petabyte sleep cheaply while remaining instantly readable?

CloudFront answers: How do we stop paying and waiting on the origin for every one of those 30 images, on every page view, worldwide?

Neither layer alone is the full story. Storage class optimization without caching leaves egress on the table. CDN without sensible storage economics still leaves a heavy monthly storage footprint.

Estimated Monthly Cost: Before vs After

The numbers below are illustrative estimates using US East (N. Virginia) public pricing. Your real bill depends on region, access patterns, and cache hit ratio. Adjust the traffic inputs to match your metrics.

Shared assumptions

Before: S3 Standard + direct S3 URLs (no CDN)

Every page view pulls all 30 images straight from S3. You pay storage, requests, and full internet egress on every byte.

Where the money goes: Storage is ~46%. Egress is ~52%. Requests are small — but that changes after moving to GIR.

After: S3 GIR + CloudFront + cache

Storage moves to GIR. Users hit CloudFront edges. S3 is only contacted on cache miss. S3 → CloudFront transfer in the same region is free.

Example assumes 90% cache hit ratio (repeat views, shared thumbnails, stable image URLs):

Estimated savings vs before: ~$8,900 / month (~36%)

How cache hit ratio changes the result

GIR charges per GB retrieved and per origin GET. CloudFront is what keeps those costs under control.

Important: GIR alone is not enough

If you move to GIR but keep serving direct S3 URLs (no CloudFront), you still pay retrieval on every read and GIR GET pricing is much higher:

That is worse than Standard + direct S3. The two-step approach matters: GIR for resting bytes, CloudFront for serving bytes.

Side-by-side summary

Pricing sources: AWS S3 and CloudFront public rates (US East). Verify at aws.amazon.com/s3/pricing and aws.amazon.com/cloudfront/pricing.

What We Learned

1. Start with the full cost stack — storage, requests, retrieval, egress — not just TB stored.

2. Match storage class to access pattern — GIR fit our "archive volume, instant read" constraint better than classic Glacier restore workflows.

3. Treat prefixes as an ops tool — essential at 500 TB, but not a substitute for CDN strategy.

4. Expect the bottleneck to shift — after storage wins, egress often becomes the next frontier.

5. Cache hit ratio is a product decision — URL stability, TTLs, and immutable assets matter as much as enabling CloudFront.

If you are running a large image surface area, the question is not only "How cheaply can we store this?" but "How often must we pull half a petabyte across the network?" Answering both — with the right storage class and the right edge cache — is how we kept costs and UX pointed in the same direction.

Quick Reference Checklist

• Map monthly spend: storage vs requests vs retrieval vs egress

• Classify objects: hot vs warm vs cold (GIR candidates = large + rarely read)

• Define prefix conventions for lifecycle and migration

• Put CloudFront (or equivalent CDN) in front of image delivery

• Version URLs + set Cache-Control for high hit ratio

• Track hit ratio, origin egress, and p95 image load time together

Have a similar workload? Start by measuring one representative page: 30 images × daily page views × average object size. That single multiplication often explains why egress, not storage, is the bill that keeps growing.