It was 2 AM. Production was on fire.
Our e-commerce platform was crawling, page loads taking 8 seconds, checkout timing out, and Slack blowing up.
We were throwing money at bigger servers. More replicas. More read nodes. Nothing worked.
Then one of our senior engineers walked in, cracked open a Red Bull, looked at the query logs for 90 seconds, and said:
“We’re fetching the same product catalogue 40,000 times per minute. Cache it.”
Twenty minutes later, response times dropped from 8 seconds to 180ms. No new servers. No rewrite. Just a cache.
That night, I learned something that changed the way I write software: the fastest code is the code that never runs. And caching is how you make that happen.
Whether you’re a junior dev just getting your footing or a staff engineer designing distributed systems, if you’re not thinking deeply about caching, you’re leaving massive performance gains on the table. This article will change that.
Let’s go deep.
What Even Is Caching?
Most textbooks define caching as “storing frequently accessed data in a fast storage layer.” Technically correct. Practically useless.
Here’s a better mental model:
Caching is the art of remembering the answer so you never have to solve the same problem twice.
Every time your app fetches a user profile from a database, renders a template, resolves a DNS query, or computes a recommendation, that’s work. Work costs time. Work costs money. Work costs scale.
Caching intercepts that work and says: “Hey, we’ve done this before. Here’s the answer.”
The difference between an app that handles 1,000 requests/second and one that handles 1,000,000? Often, it’s just how thoughtfully caching is applied.
The Cache Hierarchy You Need to Know
Before we get into strategies, let’s map the battlefield. Caches exist at every layer of your stack, and understanding where they live tells you how to use them.
User's Request
│
▼
┌─────────────┐
│ Browser │ ← HTTP Cache, Service Worker Cache
│ Cache │
└──────┬──────┘
│
▼
┌─────────────┐
│ CDN │ ← Edge Cache (Cloudflare, Fastly, etc.)
└──────┬──────┘
│
▼
┌─────────────┐
│ Reverse │ ← Nginx, Varnish, API Gateway Cache
│ Proxy │
└──────┬──────┘
│
▼
┌─────────────┐
│Application │ ← In-memory (Redis, Memcached)
│ Server │ ← Local in-process cache
└──────┬──────┘
│
▼
┌─────────────┐
│ Database │ ← Query Cache, Buffer Pool
└─────────────┘Every hop down this chain is slower and more expensive than the one above it. Your job is to answer requests as high up in this stack as possible.
Types of Caching
Let’s walk through each type, not just what they are, but when you’d actually reach for them.
1. 🖥️ In-Memory Caching (The Workhorse)
What it is: Storing data in RAM, either in your application process or in a dedicated service like Redis or Memcached.
Why it’s fast: RAM access is measured in nanoseconds. Disk access? Milliseconds. The difference is roughly the same as walking to your kitchen vs. flying to Japan to get a glass of water.
Tools: Redis, Memcached, Guava Cache (Java), lru-cache (Node.js), functools.lru_cache (Python)
Real-world use case:
import redis
import json
r = redis.Redis(host='localhost', port=6379, db=0)
def get_user_profile(user_id: str) -> dict:
cache_key = f"user:profile:{user_id}"
# Try cache first
cached = r.get(cache_key)
if cached:
return json.loads(cached)
# Cache miss - go to database
user = db.query("SELECT * FROM users WHERE id = %s", user_id)
# Store in cache for 10 minutes
r.setex(cache_key, 600, json.dumps(user))
return userWhen to use it: User sessions, API responses, computed aggregates, feature flags, rate limiting counters.
Watch out for: Memory limits, cache stampedes (more on this later), and stale data if your TTL isn’t tuned right.
2. 🌐 Distributed Caching (Caching at Scale)
What it is: A cache that spans multiple nodes, so all your application servers share the same cache layer. Redis Cluster, Memcached with consistent hashing, Hazelcast.
Why it matters: If you have 10 application servers and each has its own local cache, Server A might cache a user’s profile while Server B re-fetches it from the database. Distributed caching ensures one shared truth.
App Server 1 ──┐
App Server 2 ──┼──► Redis Cluster ──► Database
App Server 3 ──┘Real-world scenario: You’re running a flash sale. 50,000 concurrent users hit your product page. Without a distributed cache, each of your 20 app servers is independently slamming your database. With Redis Cluster, there’s one cache hit/miss decision — and 49,999 requests never touch the DB.
When to use it: Any production system with more than one application server. Honestly, just default to this.
3. 🌍 CDN Caching (The Global Cache)
What it is: Caching static and dynamic content at edge nodes geographically close to your users. Cloudflare, AWS CloudFront, Fastly, Akamai.
Why it’s game-changing: A user in Mumbai requesting your US-hosted app normally has ~180ms of network latency before your server even starts processing. With a CDN edge node in Mumbai, that drops to ~5ms.
What to cache at the CDN:
Images, videos, fonts, JS/CSS bundles (obviously)
HTML pages for logged-out users
API responses that don’t require authentication
Anything with
Cache-Control: public
Cache-Control headers — the cheat sheet:
# Cache for 1 year, immutable (perfect for hashed assets)
Cache-Control: public, max-age=31536000, immutable
# Cache for 5 minutes, revalidate after
Cache-Control: public, max-age=300, stale-while-revalidate=60
# Never cache (authenticated responses)
Cache-Control: private, no-storeThe rule: If your content is the same for all users and doesn’t change often, CDN cache it. Full stop.
4. 🗄️ Database Caching (What Your DB Is Already Doing)
What it is: The database engine itself caches things, query results, table data in memory (buffer pool), and execution plans.
PostgreSQL buffer cache in action:
-- Check what's in PostgreSQL's buffer cache
SELECT
relname,
heap_blks_read, -- physical disk reads
heap_blks_hit, -- cache hits
round(heap_blks_hit::numeric /
(heap_blks_hit + heap_blks_read) * 100, 2) AS hit_ratio
FROM pg_statio_user_tables
ORDER BY heap_blks_hit DESC;A hit ratio above 99% means your database is almost entirely serving from memory. Below 90%? Time to increase shared_buffers or rethink your query patterns.
Beyond the buffer pool: Use materialized views for expensive aggregations. Pre-compute yesterday’s analytics instead of running GROUP BY across 10M rows on every dashboard load.
-- Materialized view: compute once, read many times
CREATE MATERIALIZED VIEW daily_revenue_summary AS
SELECT
DATE(created_at) as sale_date,
SUM(amount) as total_revenue,
COUNT(*) as order_count
FROM orders
GROUP BY DATE(created_at);
-- Refresh nightly
REFRESH MATERIALIZED VIEW CONCURRENTLY daily_revenue_summary;5. 🌐 HTTP/Browser Caching (Free Performance)
What it is: The browser’s built-in cache that stores responses from previous requests.
This is the most underused cache in web development. Every time a returning visitor hits your site and their browser re-downloads your 200KB JavaScript bundle, that’s a failure. You had a chance to serve it from cache, and you didn’t take it.
ETag-based caching:
// Express.js - automatic ETag support
app.use(express.static('public', {
etag: true,
lastModified: true,
setHeaders: (res, path) => {
if (path.endsWith('.html')) {
// HTML: always revalidate
res.setHeader('Cache-Control', 'no-cache');
} else if (path.match(/\.(js|css|png|jpg|woff2)$/)) {
// Hashed assets: cache forever
res.setHeader('Cache-Control', 'public, max-age=31536000, immutable');
}
}
}));The golden rule for browser caching:
Content-addressable assets (hashed filenames)? Cache forever.
HTML files?
no-cache(always revalidate, but use the cached version if unchanged).API responses? Depends on sensitivity and freshness requirements.
6. ⚡ Application-Level / Object Caching
What it is: Caching the results of computations, not just database queries. Memoization is the purest form of this.
// Memoization — cache function results
function memoize(fn) {
const cache = new Map();
return function(...args) {
const key = JSON.stringify(args);
if (cache.has(key)) {
return cache.get(key); // Return cached result
}
const result = fn.apply(this, args);
cache.set(key, result);
return result;
};
}
// Expensive Fibonacci - now cached
const fib = memoize((n) => {
if (n <= 1) return n;
return fib(n - 1) + fib(n - 2);
});
console.log(fib(40)); // First call: computed
console.log(fib(40)); // Second call: instantReal power move: Cache entire serialized objects, not raw DB rows, but fully transformed, ready-to-serve response objects. Skip the serialization step on every request.
7. 🔄 Write-Behind / Write-Through Caching
A special type worth understanding: caches that sit between your app and your database.
Write-Through: Write to cache AND database simultaneously. Strong consistency, but writes are slower.
Write-Behind (Write-Back): Write to cache immediately, flush to database asynchronously. Blazing fast writes, but you risk data loss if the cache crashes before flushing.
This is what gaming leaderboards use. Why write every score update to PostgreSQL in real-time? Write to Redis instantly, batch-flush to the database every 30 seconds.
Caching Strategies
Knowing the types is table stakes. The strategies are where senior engineers earn their keep.
A caching strategy answers two questions:
How does data get into the cache? (population)
What happens when it’s no longer valid? (invalidation)
Strategy 1: Cache-Aside (Lazy Loading) — The Default
The pattern:
Read:
1. Check cache
2. Cache HIT → return data ✓
3. Cache MISS → fetch from DB → write to cache → return data
Write:
1. Update database
2. Invalidate (delete) the cache entrydef get_product(product_id: str) -> dict:
cache_key = f"product:{product_id}"
# 1. Check cache
product = cache.get(cache_key)
if product:
return product # Cache HIT
# 2. Cache MISS — fetch from source
product = db.find_product(product_id)
# 3. Populate cache
cache.set(cache_key, product, ttl=3600)
return product
def update_product(product_id: str, data: dict):
db.update_product(product_id, data)
cache.delete(f"product:{product_id}") # InvalidateWhy it’s the default: Simple, resilient (app works fine if cache is down), and only caches what’s actually requested.
The downside: First request after a cache miss (or cold start) is always slow. And under high concurrency, you can get a cache stampede, 1,000 requests all miss the cache simultaneously and all hammer the database at once.
💡 Enjoying this article?
Every week day, I publish practical, production-ready deep dives covering Web development, System Design, Open source projects, Tech industry trends and AI Engineering and tools.
Strategy 2: Read-Through Caching — The Cache Takes Control
The difference from Cache-Aside: The application never talks to the database directly. The cache library handles the miss automatically.
// Using a read-through cache library
const cache = new ReadThroughCache({
loader: async (key) => {
// This runs automatically on cache miss
const userId = key.replace('user:', '');
return await db.users.findById(userId);
},
ttl: 600
});
// Application code is beautifully simple
async function getUser(userId) {
return await cache.get(`user:${userId}`);
// Cache handles hit/miss/load automatically
}When to use it: When you want a clean separation of concerns and don’t want cache logic scattered through your application code.
Strategy 3: Write-Through Caching — Keep Cache Fresh on Every Write
The pattern: Every write goes to both cache and database synchronously.
def save_user_preferences(user_id: str, prefs: dict):
# Write to both simultaneously
db.update("UPDATE prefs SET data = %s WHERE user_id = %s",
json.dumps(prefs), user_id)
cache.set(f"prefs:{user_id}", prefs, ttl=86400)
# Cache is guaranteed fresh after thisThe tradeoff: Writes are slightly slower (two operations), but reads are always fast. No stale data. No cache misses on hot paths.
Best for: User preferences, configuration data, anything that’s read far more than it’s written.
Strategy 4: Write-Behind (Write-Back) Caching — Maximum Write Performance
The pattern: Write to cache immediately, database gets updated asynchronously.
import asyncio
from collections import defaultdict
class WriteBehindCache:
def __init__(self, flush_interval=30):
self.cache = {}
self.dirty_keys = set()
self.flush_interval = flush_interval
asyncio.create_task(self._flush_loop())
async def set(self, key: str, value):
self.cache[key] = value
self.dirty_keys.add(key) # Mark as needing DB write
# Return immediately - don't wait for DB
async def _flush_loop(self):
while True:
await asyncio.sleep(self.flush_interval)
await self._flush_dirty_keys()
async def _flush_dirty_keys(self):
keys_to_flush = self.dirty_keys.copy()
self.dirty_keys.clear()
# Batch write to database
batch = {k: self.cache[k] for k in keys_to_flush if k in self.cache}
await db.batch_update(batch)Where this shines: Gaming leaderboards, real-time analytics counters, activity feeds, and any write-heavy workload where microseconds matter.
The risk: If your cache crashes before flushing, you lose writes. Always have a durability strategy (Redis persistence, WAL logging, etc.).
Strategy 5: Refresh-Ahead (Proactive Caching) — Eliminate Cache Miss Latency
The insight: If you know a cache entry is about to expire, and you know it’ll be requested again — why wait for the miss? Refresh it before it expires.
import threading
import time
class RefreshAheadCache:
def __init__(self, ttl=600, refresh_threshold=0.8):
self.store = {}
self.ttl = ttl
# Refresh when 80% of TTL has elapsed
self.refresh_threshold = refresh_threshold
def get(self, key: str, loader_fn):
entry = self.store.get(key)
if entry:
age = time.time() - entry['cached_at']
# Proactively refresh in background if nearing expiry
if age > (self.ttl * self.refresh_threshold):
threading.Thread(
target=self._background_refresh,
args=(key, loader_fn)
).start()
return entry['value'] # Still return current value
# True cache miss - load synchronously
value = loader_fn(key)
self._store(key, value)
return value
def _background_refresh(self, key, loader_fn):
value = loader_fn(key)
self._store(key, value)
def _store(self, key, value):
self.store[key] = {'value': value, 'cached_at': time.time()}The magic: Users never experience a stale cache miss. The cache silently refreshes itself in the background. This is how Netflix keeps their homepage blazing fast — predictive prefetching based on what you’re likely to request next.
Strategy 6: Cache Stampede Prevention — The Thundering Herd Problem
This one will save you from 3 AM incidents.
The scenario: A hot cache key expires. At that exact millisecond, 5,000 concurrent requests all get a cache miss. All 5,000 hit your database simultaneously. Your database falls over. Your app is down.
Solution 1: Probabilistic Early Expiration (XFetch)
import math
import random
import time
def get_with_stampede_prevention(key: str, loader_fn, ttl=600, beta=1.0):
entry = cache.get_with_ttl(key)
if entry:
value, remaining_ttl = entry
# XFetch algorithm: probabilistically expire early
# Higher beta = more aggressive early refresh
if -beta * math.log(random.random()) >= remaining_ttl:
# Recompute early to prevent future stampede
value = loader_fn(key)
cache.set(key, value, ttl)
return value
return loader_fn(key)Solution 2: Mutex Lock (only one request rebuilds the cache)
import redis
def get_with_lock(key: str, loader_fn, ttl=600):
r = redis.Redis()
lock_key = f"lock:{key}"
# Check cache
value = r.get(key)
if value:
return json.loads(value)
# Try to acquire lock
lock_acquired = r.set(lock_key, "1", nx=True, ex=10)
if lock_acquired:
try:
# This process rebuilds the cache
value = loader_fn(key)
r.setex(key, ttl, json.dumps(value))
return value
finally:
r.delete(lock_key)
else:
# Another process is rebuilding - wait briefly and retry
time.sleep(0.1)
return get_with_lock(key, loader_fn, ttl)Cache Eviction Policies: When the Cache Is Full
When your cache hits capacity, it needs to decide what to remove. This decision matters more than most engineers realize.
Redis eviction policy cheat sheet:
allkeys-lru → General web app cache (recommended default)
volatile-lru → When only TTL-set keys should be evicted
allkeys-lfu → When some keys are dramatically more popular
noeviction → When data loss is unacceptable (use with caution)Cache Invalidation: The Hardest Problem in Computer Science
Phil Karlton famously said: “There are only two hard things in Computer Science: cache invalidation and naming things.”
He wasn’t joking.
Strategy 1: TTL-Based (Simple, Blunt) Set a time limit. Accept that data may be stale for up to TTL seconds. Best for: Data that doesn’t need to be perfectly fresh.
Strategy 2: Event-Driven Invalidation (Surgical) When data changes, explicitly invalidate or update the cache.
# Using Redis pub/sub for distributed cache invalidation
def on_product_updated(product_id: str):
# Update the cache immediately
updated_product = db.get_product(product_id)
cache.set(f"product:{product_id}", updated_product, ttl=3600)
# Broadcast invalidation to all app servers
redis_pubsub.publish('cache_invalidation', json.dumps({
'type': 'product',
'id': product_id
}))
# All app servers listen and invalidate their local caches
def handle_invalidation_message(message):
data = json.loads(message['data'])
local_cache.delete(f"{data['type']}:{data['id']}")Strategy 3: Cache Versioning (Nuclear Option) Embed a version number in your cache key. Want to invalidate everything? Just bump the version.
CACHE_VERSION = "v3"
def cache_key(resource_type: str, resource_id: str) -> str:
return f"{CACHE_VERSION}:{resource_type}:{resource_id}"
# Invalidate ALL caches globally:
# Just change CACHE_VERSION = "v4"
# Old keys become orphaned (will naturally expire)The Caching Mindset: A Decision Framework
Before you cache anything, ask these five questions:
1. How often does this data change?
Almost never → Aggressive caching, long TTL
Every few minutes → Moderate TTL, event-driven invalidation
Every second → Don’t cache (or cache with subsecond TTL + accept staleness)
2. How expensive is a cache miss?
DB query in <5ms → Maybe don’t bother
Complex aggregation taking 2s → Absolutely cache it
3. How bad is stale data?
Product description → Totally fine to be 5 minutes stale
Account balance → Never serve stale. Never.
4. How many users share this data?
Same for all users → Cache at CDN/shared layer
Per-user → Cache in user-specific namespace
5. What’s the read-to-write ratio?
Read-heavy (100:1+) → Caching has massive ROI
Write-heavy (1:1) → Caching overhead may not be worth it
Common Caching Anti-Patterns (Learn From Others’ Pain)
❌ Caching Everything
Not everything benefits from caching. Caching data that changes every 100ms adds complexity with zero benefit.
❌ Cache Stampede on Cold Start
When you deploy fresh servers with empty caches, all traffic hits the database simultaneously. Solution: Warm your caches during deployment before routing traffic.
❌ Stale Cache Forever
Forgetting TTLs entirely. One day, you’re debugging why users see data from 6 months ago. Solution: Every cache entry gets a TTL. Always.
❌ Caching Personalized Data at the CDN
Caching a “Welcome, John” response at the CDN means every user gets “Welcome, John.” Solution: Separate public and private content clearly.
❌ Not Monitoring Cache Hit Rates
A cache with a 40% hit rate is barely helping you. Solution: Track cache hits, misses, and evictions as first-class metrics.
# Track cache metrics
class InstrumentedCache:
def __init__(self):
self.hits = 0
self.misses = 0
def get(self, key):
value = self._backend.get(key)
if value:
self.hits += 1
metrics.increment('cache.hit', tags=[f'key_prefix:{key.split(":")[0]}'])
else:
self.misses += 1
metrics.increment('cache.miss', tags=[f'key_prefix:{key.split(":")[0]}'])
return value
@property
def hit_rate(self):
total = self.hits + self.misses
return self.hits / total if total > 0 else 0Putting It All Together: Real Architecture Example
Here’s how a production e-commerce product page might layer caching:
User Request: GET /product/abc123
│
▼
┌─────────────────────┐
│ CDN Edge Cache │ ← Cache HTML for 60s for anonymous users
│ (Cloudflare) │ Cache-Control: public, max-age=60
└──────────┬──────────┘
│ MISS
▼
┌─────────────────────┐
│ Load Balancer / │ ← Rate limiting counters in Redis
│ API Gateway │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Application Layer │ ← Cache-Aside with Redis
│ (Node.js / Python) │ Key: product:abc123, TTL: 3600s
└──────────┬──────────┘
│ MISS
▼
┌─────────────────────┐
│ PostgreSQL │ ← shared_buffers: 8GB
│ (with hot buffer) │ Frequently accessed tables in memory
└─────────────────────┘Result: ~95% of requests never hit the database. The 5% that do are fast because the DB’s own buffer cache is warm.
Final Thoughts
The engineers I’ve seen fail at caching didn’t fail because they didn’t know what a cache was. They failed because they treated caching as an afterthought, bolted on after things got slow, without a strategy, without monitoring, without thinking through invalidation.
The engineers I’ve seen win with caching thought about it like a first-class system design concern. They asked the right questions upfront. They chose the right strategy for each use case. They monitored hit rates obsessively. They planned for invalidation before writing the first line of cache code.
The real lesson from that 2 AM incident? It wasn’t that caching saved us. It’s what we should have been caching from day one. The product catalogue hadn’t changed in weeks. We were recomputing the same result 40,000 times per minute for no reason.
Every cache miss is a question you’ve already answered, that you’re making yourself answer again.
Stop answering the same questions twice. Cache intentionally.
Thank You for Reading!
I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.