Concepts for
Scalable Applications

1.1

What "Scalable" Actually Means

A system is scalable if it can handle growth in load by adding resources, with performance remaining acceptable and cost growing no faster than load. Notice what that definition does not say: it does not say "fast". A system can be fast for one user and unscalable; it can be slow and perfectly scalable.

The four related -ilities

Describing load: load parameters

"More load" is vague. Before you can scale, you must name the load parameters that actually stress your system. These differ wildly between applications:

• Requests per second to the web tier (a news site at breaking-news time).
• Ratio of reads to writes (a social feed is read 100× more often than written).
• Number of simultaneously active users or open connections (a chat app with long-lived WebSockets).
• Data volume and its growth rate (an analytics platform ingesting events).
• Fan-out — how much work one action triggers. When a celebrity with 50 million followers posts, does the system write 50 million timeline entries, or compute the timeline at read time? Twitter famously re-architected around exactly this question.

1.2

Measuring Performance: Latency, Throughput & Percentiles

If you serve 1 million requests, you have 1 million latency measurements. They are never all the same: a cache miss, a garbage-collection pause, a lost network packet or a noisy neighbor on the same host will make some requests far slower than others. So latency is a distribution, not a number — and how you summarize that distribution determines whether you see your problems or hide them.

Why averages lie

Imagine 99 requests take 10 ms and 1 request takes 5,000 ms. The average is ~60 ms — a number that describes no actual request. It is too pessimistic for the 99 and wildly optimistic for the 1. Worse, that one user waited five seconds and probably left.

Instead we use percentiles. Sort all latencies from fastest to slowest: the p50 (median) is the value halfway down; the p95 is the value 95% of the way down, meaning 95% of requests were faster and 5% slower; p99 and p99.9 describe the slow tail.

Why the tail matters more than you think

It's tempting to dismiss p99 as "only 1% of requests". Three facts make tail latency critical:

1. Your most valuable users hit it most. Users with the most data (largest carts, longest histories) tend to have the slowest requests — and are often your best customers.
2. One page = many requests. If a page load makes 20 API calls, the chance that at least one lands in the slowest 1% is 1 − 0.99²⁰ ≈ 18%. Your p99 becomes the experience of nearly one in five page views.
3. Fan-out amplifies it. If a backend request fans out to 100 servers in parallel and must wait for all of them, its latency is the maximum of 100 samples. Then 1 − 0.99¹⁰⁰ ≈ 63% of requests experience at least one server's p99. This "tail at scale" effect is why companies like Google invest heavily in hedged requests and tail-tolerant design.

Throughput, and its relationship to latency

Throughput is completed work per unit time. At low load, latency and throughput are independent: doubling the arrival rate doubles throughput and latency stays flat. But as you approach a resource's capacity, requests start waiting for each other, and latency climbs — at first slowly, then explosively. Understanding why requires a little queueing theory, which is next.

1.3

Queueing Theory, Little's Law & the Utilization Trap

Little's Law

The most useful equation in capacity planning is astonishingly simple. For any stable system:

It holds regardless of arrival distribution or service order, which makes it incredibly practical. Worked examples:

• Concurrency from traffic. An API receives 2,000 requests/second and each takes 150 ms on average. At any moment there are 2,000 × 0.15 = 300 requests in flight. If your server framework handles requests with one thread each and has a pool of 200 threads, you will queue — you need more threads, more instances, or lower latency.
• Sizing a database connection pool. Those same 2,000 req/s each hold a DB connection for 5 ms on average. Connections busy = 2,000 × 0.005 = 10. With 5 app instances, a pool of ~4–6 each (plus headroom) is plenty. Teams routinely configure pools of 100 per instance, which can overwhelm the database with idle connections when they scale out to 50 instances.
• Working backwards. A queue consumer processes 500 messages/second and you observe ~30,000 messages backlogged. Each new message waits about 30,000 ÷ 500 = 60 seconds.

Utilization vs latency: the hockey stick

For the simplest queueing model (random arrivals, one server — "M/M/1"), the average time in the system is W = S / (1 − ρ), where S is the service time and ρ (rho) is utilization: the fraction of time the server is busy. At 50% utilization a request takes 2× its service time on average; at 90% it takes 10×; at 95%, 20×.

The practical consequences are profound:

• "CPU at 90% is efficient" is a trap. For latency-sensitive services, teams usually target 50–70% peak utilization and treat the rest as headroom that buys stable latency and room to absorb spikes or a lost server.
• Small load increases near saturation cause huge latency increases. Going from 85% to 95% utilization (+12% load) triples waiting time in this model. This is why outages often seem to "come out of nowhere".
• Reducing variance helps. Random arrivals and variable service times cause queues. Smoothing traffic (queues, rate limits) and eliminating occasional slow operations both pull the curve down.

Interactive: Little's Law calculator

1.4

Scaling Laws: Amdahl's Law & the Universal Scalability Law

If one server handles 1,000 req/s, do ten servers handle 10,000? Sometimes. Usually not quite. Occasionally, adding servers makes things worse. Two models explain why.

Amdahl's Law: the serial fraction

Any workload has a part that can run in parallel and a part that cannot — writing to one primary database, acquiring a global lock, a single coordinator. If a fraction p is parallelizable, the best speedup with N workers is:

With p = 0.95, infinite servers give at most 1 / 0.05 = 20×. A mere 5% serial portion caps you at 20× no matter how much money you spend. That is why engineers obsess over eliminating shared bottlenecks.

The Universal Scalability Law (USL)

Neil Gunther's USL adds a second penalty: coherency (crosstalk) — the cost of nodes coordinating with each other (cache invalidation, distributed locks, gossip, replication acknowledgments). Contention (α) causes diminishing returns; coherency (β) grows with N², so beyond a peak, adding nodes reduces throughput.

Design lesson: scalable architectures minimize shared things — shared locks, shared databases, synchronous cross-node chatter. Partitioning (giving each node its own independent slice of work and data) is the most powerful tool for pushing α and β toward zero; you'll see it again in caching, sharding and queue partitions.

1.5

Vertical vs Horizontal Scaling

There are exactly two ways to add capacity, and every architecture is a blend of both.

1.6

Stateless vs Stateful Services

Horizontal scaling only works if any instance can serve any request. That requires the web tier to be stateless: it may cache data, but it must not hold anything in local memory or on local disk that a future request from the same user depends on.

The usual suspects: state hiding in your app

• In-memory sessions (the default for many frameworks — e.g. express-session's MemoryStore).
• Uploaded files written to local disk — the next request goes to another server and the file "disappears".
• In-process caches that must be consistent across users (fine as an optimization, dangerous if correctness depends on it).
• Scheduled jobs / cron inside the web process — with 10 replicas, the "daily email" is sent 10 times.
• WebSocket connections — inherently tied to one server; broadcasts need a shared pub/sub (e.g. Redis) between instances.

Three ways to handle sessions

1. Sticky sessions (session affinity). The load balancer pins a user to one server via a cookie. Quick fix, but load becomes uneven, deployments log people out, and a crash loses data. Acceptable only as a stopgap.
2. Centralized session store. Sessions live in Redis (or a database). Every server reads them by session ID. Simple and robust; costs ~0.5 ms per request and makes the store a dependency that must itself be highly available.
3. Client-side tokens (e.g. signed JWTs or encrypted cookies). The state travels with the request, signed so it can't be tampered with. Zero server lookup — but tokens can't easily be revoked before they expire, and they grow every request's size. Often paired with short expiry plus a revocation list.

Lab: break it, then fix it

Add this naïve visit counter to server.js. It stores state in process memory:

// ❌ STATEFUL — each replica has its own private Map
const visits = new Map();
app.get('/api/visits/:user', (req, res) => {
  const n = (visits.get(req.params.user) || 0) + 1;
  visits.set(req.params.user, n);
  res.json({ user: req.params.user, visits: n, instance: INSTANCE });
});

docker compose up -d --build --scale app=3
for i in $(seq 1 6); do curl -s localhost:8080/api/visits/alice | jq -c; done
# {"user":"alice","visits":1,"instance":"a1b2…"}
# {"user":"alice","visits":1,"instance":"c3d4…"}   ← different replica, count reset!
# {"user":"alice","visits":1,"instance":"e5f6…"}
# {"user":"alice","visits":2,"instance":"a1b2…"}

Now move the state into Redis, which all replicas share:

// ✅ STATELESS — the counter lives in Redis; INCR is atomic, so
// concurrent requests on different replicas never lose an update.
app.get('/api/visits/:user', async (req, res, next) => {
  try {
    const n = await cache.incr(`visits:${req.params.user}`);
    res.json({ user: req.params.user, visits: n, instance: INSTANCE });
  } catch (err) { next(err); }
});

Rebuild and repeat the loop: the count now increases monotonically no matter which instance answers.

1.7

Finding Bottlenecks & the Scaling Journey

A system's throughput is limited by its single most constrained resource — just as a highway's capacity is set by its narrowest bridge. Adding capacity anywhere else achieves nothing. The skill of scaling is therefore primarily a skill of diagnosis.

Two methods that find bottlenecks fast

Saturation is the key signal. A resource at 100% utilization with no queue is fine; a resource at 70% with a growing queue is your bottleneck. Remember too that bottlenecks are often soft limits: a connection pool of 10, a thread pool of 200, a file-descriptor limit of 1,024, a Node.js event loop blocked by synchronous code.

Lab: find the bottleneck in scale-lab

The /api/cpu endpoint burns CPU synchronously, blocking Node's single-threaded event loop. Load it with an open arrival rate:

// k6/cpu.js
import http from 'k6/http';
export const options = {
  scenarios: {
    open_model: {
      executor: 'constant-arrival-rate',
      rate: 40, timeUnit: '1s',            // 40 new requests every second, no matter what
      duration: '30s',
      preAllocatedVUs: 50, maxVUs: 500,
    },
  },
};
export default function () { http.get('http://localhost:8080/api/cpu?ms=20'); }

docker compose up -d --scale app=1 && k6 run k6/cpu.js   # one replica: ρ ≈ 40 × 0.020 = 0.8
docker compose up -d --scale app=3 && k6 run k6/cpu.js   # three replicas: ρ ≈ 0.27 each
docker stats --no-stream                                    # watch CPU % per container

With one replica, utilization is ~80% — right on the steep part of the hockey stick — so p95 is many times the 20 ms service time. With three replicas each runs at ~27% and p95 falls close to 20 ms. Try rate: 60 with one replica (ρ = 1.2): latency grows throughout the test and requests start timing out. You have just reproduced Figure 1.2 on your own machine.

Interactive: the scaling journey from 1 to millions of users

Most successful systems evolve through remarkably similar stages. Step through them — each stage removes the bottleneck exposed by the previous one.

1.8

Availability, the "Nines" & SLOs

Scale and reliability are inseparable: more machines means more things failing every day. At a thousand servers, a component with a 3-year mean time between failures breaks almost daily. Large systems are designed on the assumption that something is always broken.

The nines

Each extra nine costs roughly an order of magnitude more engineering effort. At 99.99% there is no time for a human to even notice an incident before the monthly budget is gone — recovery must be automatic.

Series vs parallel: the arithmetic of dependencies

• In series (A calls B calls C — all must work): availabilities multiply. Three 99.9% components give 0.999³ ≈ 99.7%. Every new hard dependency makes you less available.
• In parallel (redundant replicas, any one suffices): unavailabilities multiply. Two independent 99% replicas give 1 − 0.01² = 99.99%. This is why redundancy is so powerful — as long as failures are truly independent (different hosts, racks, availability zones).

SLI, SLO, SLA and error budgets

• SLI (indicator) — what you measure: "the proportion of HTTP requests that return non-5xx in under 300 ms".
• SLO (objective) — your internal target for the SLI: "99.9% over a rolling 30 days".
• SLA (agreement) — a contract with customers, with penalties (refunds, credits). Always looser than the SLO so you have a buffer.
• Error budget — 100% minus the SLO. A 99.9% SLO gives you 0.1% (≈43 minutes/month) to "spend" on risky deploys, experiments and failures. When the budget is exhausted, teams freeze features and invest in reliability. It turns the eternal "speed vs stability" argument into arithmetic.

A first look at CAP and PACELC

Once data is replicated across machines, the network between them can fail. The CAP theorem says that during a network Partition, a replicated data system must choose between Consistency (every read sees the latest write, or errors) and Availability (every request gets a non-error answer, possibly stale). PACELC extends it: else, when the network is healthy, you still trade Latency against Consistency, because waiting for replicas to agree takes time. We'll study this properly in the database module; for now, internalize that replication is never free — you always pay in latency, consistency, or availability.

1.9

Back-of-the-Envelope Capacity Planning

Senior engineers can estimate, in five minutes and on a napkin, whether a design needs 3 servers or 300. The goal is not precision — it's being within an order of magnitude, fast, so obviously wrong designs are discarded early. Let's work a complete example.

1. Read traffic. 10 M × 50 = 500 M photo views/day. ÷ ~10⁵ s/day ≈ 5,000 req/s average. Assume peak is 3× the average → ~15,000 req/s at peak.
2. Write traffic. 10 M × 0.2 = 2 M uploads/day ≈ 23 uploads/s average, ~70/s peak. Read:write ratio ≈ 250:1 → this is a read-heavy system; caching and CDNs will dominate the design.
3. App servers. Each instance does 800 req/s at max; we target 60% utilization (the hockey stick!) → 480 req/s usable. 15,000 ÷ 480 ≈ 32 instances. Spread across 3 availability zones and require surviving the loss of one zone: the remaining two zones must carry the full peak, so we need 32 × 3/2 = 48 instances (16 per zone).
4. But wait — a CDN. Photos are immutable once uploaded, which makes them perfect for CDN caching. If 95% of views are CDN hits, the origin sees only 750 req/s at peak: 2–3 instances per zone. One design choice just cut the fleet by ~85%.
5. Storage. 2 M × 300 KB = 600 GB/day ≈ 220 TB/year. With 3× replication (or ~1.5× with erasure coding in object storage), plan for a few hundred TB to ~0.7 PB per year — clearly object storage (S3-like), not a database.
6. Bandwidth. 15,000 views/s × 300 KB ≈ 4.5 GB/s ≈ 36 Gbit/s at peak — again, a CDN's job, not your servers'.

1.10

Key Takeaways

Next up, Module 02 — Load Balancing & Reverse Proxies: how traffic is actually spread across those stateless replicas, from DNS to L4 and L7 balancers, algorithms, health checks and connection draining.

Checkpoint

Module 01 Quiz

This one is longer — it covers every concept from the module, including the math. Use the calculators above if you need them.

2.1

What a Load Balancer Really Does

In Module 01 we made the application tier stateless so that any replica can serve any request. That raises an obvious question: who decides which replica gets which request? That's the job of a load balancer (LB). It sits between clients and a pool of servers and presents the whole pool as a single address.

Distributing traffic is only the most visible of its jobs. A production LB typically does all of the following:

Load balancer vs reverse proxy vs API gateway

These terms overlap heavily, and the same software (Nginx, HAProxy, Envoy, Traefik, cloud LBs) can play all three roles. A reverse proxy is any server that accepts client requests on behalf of backends and forwards them — even to a single backend. A load balancer is a reverse proxy (or a packet forwarder) whose main purpose is spreading load across many backends. An API gateway is a reverse proxy specialized for APIs, adding authentication, rate limiting, request transformation and API keys. When people say "the load balancer" in a modern web stack, they usually mean an L7 reverse proxy.

2.2

Where to Balance: DNS, Layer 4 and Layer 7

Load balancing can happen at several layers of the network stack, and the layer determines what the balancer can see — and therefore what it can decide. The numbers come from the OSI model: Layer 4 is the transport layer (TCP/UDP), Layer 7 is the application layer (HTTP).

2.3

Balancing Algorithms

Once a balancer knows which backends are healthy, it needs a rule for picking one. The choice matters more than most people expect, because real requests aren't uniform: some take 2 ms and some take 2 seconds, and some servers are slower than others (a noisy neighbor, a garbage-collection pause, an older instance type).

Why "least connections" beats round robin

Round robin assumes every request costs the same. When one server is slow, round robin keeps feeding it exactly its share, so its queue grows — and every request that lands there inherits the wait. Least connections notices that the slow server has more requests in flight and naturally sends it less. It's a feedback loop: in-flight count is a live measure of how fast each server is draining its work (Little's Law again — in-flight = rate × latency).

The power of two random choices

In large systems there isn't one balancer — there are dozens of proxy instances, or every client does its own balancing. Each has only a partial, slightly stale view of backend load. If they all run strict least-connections, they all spot the same least-loaded server at the same moment and stampede it — a herd effect. The elegant fix: pick two servers at random and send the request to the less loaded of the two. This tiny bit of randomness avoids herding, yet the maximum load drops dramatically compared to pure random (a classic result from Michael Mitzenmacher's research). Envoy, Linkerd, HAProxy and NGINX all offer it.

Interactive: simulate the algorithms

This simulator sends 20,000 requests with random (exponential) service times to 8 servers, each processing one request at a time. Raise the utilization and add a slow server to see which algorithms stay calm.

2.4

Hashing, Affinity & Consistent Hashing

Sometimes you want the same key to land on the same server every time: a cache server that holds user 42's data, a shard that owns a chat room, a server holding a WebSocket session. The naive approach is server = hash(key) % N. It works perfectly — until N changes.

With modulo hashing, going from 4 to 5 servers changes the mapping of about 80% of keys (in general, a fraction N/(N+1)). For a cache tier, that means 80% of lookups suddenly miss and fall through to the database — exactly the moment you were adding capacity because load was high.

Virtual nodes

With only three points on the ring, the arcs are uneven by chance, so one server might own 50% of the keys. The standard fix is virtual nodes: each physical server is hashed onto the ring 100–200 times (A#1, A#2, …). The many small arcs average out, giving each server close to its fair share, and when a server leaves, its keys spread across all remaining servers instead of dumping onto one neighbor. Weighted servers simply get more virtual nodes.

Affinity vs balance

Every form of affinity trades balance for locality. Hashing on client IP breaks when thousands of users share one corporate NAT address; hashing on a celebrity's user ID makes one server hot. Use affinity when locality buys something real (cache hits, in-memory shards), and pair it with bounded loads (a variant that spills keys to the next server once one exceeds, say, 125% of average) to protect against hot spots.

2.5

Health Checks, Failover & Graceful Draining

A balancer is only as good as its knowledge of which backends are healthy. Sending traffic to a dead server turns a partial failure into user-visible errors; removing a healthy server too eagerly turns a blip into a capacity crisis.

Active vs passive checks

• Active health checks: the balancer probes each backend on a schedule (e.g. GET /health every 5 s). After N consecutive failures it marks the backend down; after M consecutive successes it brings it back. Detection time ≈ interval × failure threshold.
• Passive health checks (outlier detection): the balancer watches real traffic. If a backend returns errors or times out repeatedly, it's ejected for a cool-down period. Fast to react, needs no extra requests, but only works for backends receiving traffic. Open-source Nginx supports passive checks (max_fails, fail_timeout); HAProxy, Envoy and cloud LBs do both.

What should a health endpoint check?

This is subtler than it looks. Kubernetes popularized splitting it in two:

• Liveness — "is this process fundamentally broken and in need of a restart?" (e.g. deadlocked event loop). Keep it trivial; it must not check dependencies.
• Readiness — "should this instance receive traffic right now?" (finished warming up, has DB connections, isn't shutting down).

Graceful shutdown and connection draining

Every deploy, scale-in and autoscaling event removes instances. Done carelessly, each removal kills the requests in flight on that instance — producing a burst of 502s on every deploy. Done properly, the instance is drained: it stops receiving new work, finishes what it has, then exits.

Lab: graceful shutdown in scale-lab

Replace the bottom of app/server.js with this:

let shuttingDown = false;

// Readiness: tells the load balancer whether to send us new traffic
app.get('/ready', (req, res) => {
  if (shuttingDown) return res.status(503).json({ ready: false });
  res.json({ ready: true, instance: INSTANCE });
});

async function main() {
  await cache.connect();
  const server = app.listen(PORT, () => console.log(`scale-lab ${INSTANCE} listening on ${PORT}`));

  process.on('SIGTERM', () => {
    console.log('SIGTERM: draining');
    shuttingDown = true;                       // ① fail readiness

    setTimeout(() => {                         // ② give the LB time to notice
      server.close(async () => {               // ③ stop accepting; callback fires when in-flight requests finish
        await Promise.allSettled([db.end(), cache.quit()]);
        console.log('drained, exiting');
        process.exit(0);
      });
      server.closeIdleConnections();           // drop idle keep-alive sockets (Node 18.2+)
    }, Number(process.env.DRAIN_DELAY_MS || 5000));

    setTimeout(() => process.exit(1), 25_000).unref();   // ④ hard deadline
  });
}
main();

Docker sends SIGTERM on docker compose stop but only waits 10 seconds before SIGKILL. Give the app more room in docker-compose.yml:

  app:
    # …existing settings…
    stop_grace_period: 30s

2.6

Reverse Proxy Essentials: TLS, Headers, Timeouts & Buffering

TLS termination

Terminating TLS at the proxy means certificates live in one place, the expensive handshake happens once at the edge, and the proxy can inspect HTTP to route it. Traffic from proxy to backend may then be plain HTTP on a trusted private network, or re-encrypted (often with mutual TLS) when you don't trust the network — the default assumption in zero-trust environments and for regulated data. TLS passthrough (L4) is the alternative when the backend must hold the keys.

Preserving the client's identity

Once a proxy sits in front of your app, every request appears to come from the proxy's IP. Proxies therefore add headers: X-Forwarded-For (the chain of client IPs), X-Forwarded-Proto (was the original request HTTPS?), X-Forwarded-Host, or the standardized Forwarded header. At L4, the PROXY protocol carries the same information in front of the TCP stream.

Timeouts: the most important lines in the config

A proxy without deliberate timeouts will happily hold thousands of connections open waiting on a hung backend, until it runs out of memory or file descriptors. Every hop needs three timeouts: connect (establishing the TCP connection — should be short, ~1–5 s), read/idle (gap between bytes from the backend), and an overall request deadline. Timeouts must also shrink as you go deeper: if the edge gives up after 30 s, a backend still working at second 45 is pure waste.

Retries at the proxy

Proxies can retry a failed request on another backend (Nginx's proxy_next_upstream). This hides individual instance failures beautifully — but only for idempotent requests (Nginx won't retry a POST by default), and only with a limit. Unbounded retries at several layers multiply: 3 retries at the edge × 3 at the gateway × 3 in the service = up to 27 attempts hitting an already-struggling database. Prefer retry budgets (e.g. retries may add at most 10% extra load).

Buffering and slow clients

A user on a poor mobile connection might take 10 seconds to upload a request or download a response. If your app server's worker waits the whole time, slow clients can exhaust it — the basis of the Slowloris attack. A buffering reverse proxy reads the full request from the slow client first, sends it to the backend in one fast burst, takes the response just as quickly, and then trickles it out to the client itself. The app's worker is freed in milliseconds. This is one of the oldest and most valuable reasons to put Nginx in front of an app server.

A production-shaped Nginx config

upstream app_servers {
  least_conn;                                  # balance by in-flight requests
  server app:3000 max_fails=3 fail_timeout=10s; # passive health check per resolved IP
  keepalive 64;                                # reuse connections to backends
}

server {
  listen 443 ssl;
  http2 on;
  server_name shop.example.com;
  ssl_certificate     /etc/nginx/certs/fullchain.pem;
  ssl_certificate_key /etc/nginx/certs/privkey.pem;

  client_max_body_size 10m;         # reject huge uploads early
  client_body_timeout   10s;        # slow-client protection
  client_header_timeout 10s;

  location / {
    proxy_pass http://app_servers;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_connect_timeout 2s;
    proxy_read_timeout    15s;
    proxy_next_upstream error timeout http_502 http_503;
    proxy_next_upstream_tries 2;    # at most one retry on another server
  }
}

2.7

Who Balances the Balancer? High Availability for the LB Tier

We put a load balancer in front of our servers to remove single points of failure — and promptly created a new one. The load balancer itself must be redundant. There are three common patterns, often stacked together.

1. Active-passive with a floating IP. Two balancers share a virtual IP. The active one holds it; the passive one monitors it with a protocol like VRRP (implemented by keepalived). If the active dies, the passive claims the IP within a second or two. Simple, but half your balancer capacity sits idle.
2. Active-active with ECMP. Several L4 balancers announce the same IP to the network's routers, which use Equal-Cost Multi-Path routing to hash each flow to one of them. Capacity scales out, and a failed balancer's flows move to the others. Consistent hashing in the balancers ensures a flow keeps reaching the same backend even when it switches balancer.
3. Anycast across locations. The same IP is announced via BGP from many data centers worldwide; internet routing delivers each user to the nearest one. If a site goes dark, its announcements are withdrawn and traffic flows elsewhere. CDNs and large DNS providers are built this way.

In the cloud, you rarely build these layers yourself: a managed load balancer (AWS ALB/NLB, Google Cloud Load Balancing, Azure Load Balancer / Front Door) is internally a redundant, autoscaled fleet much like Figure 2.4. Your responsibility shifts to configuring it well — health checks, timeouts, draining delays — and to spreading your backends across multiple availability zones.

2.8

Global Traffic Management

Once you run in more than one region, a new question appears: which region should this user reach? This is global server load balancing (GSLB), and it has two main tools.

• DNS-based steering. The authoritative DNS server returns different answers depending on the resolver's location (geo routing), measured latency (latency-based routing), configured percentages (weighted routing, useful for migrating traffic gradually), or health (failover routing). Its weakness is caching: answers live as long as their TTL, some resolvers and clients ignore low TTLs, and the DNS server sees the resolver's location, not the user's (the EDNS Client Subnet extension partly fixes this).
• Anycast. One IP address, announced from every location. Routing takes users to the nearest site with no DNS caching delay, and failover happens at network speed. The trade-off: the "nearest" site by network routing isn't always the lowest-latency or least-loaded one, and long-lived TCP connections can break if routing shifts mid-connection.

Many large sites combine both: anycast to reach a nearby edge PoP quickly, then the edge forwards to the best backend region over the provider's private network.

2.9

Lab: Watch the Algorithms Under Load

Let's compare round robin and least connections on real containers using the /api/cpu endpoint, with a mix of fast and slow requests.

1. Mixed workload

// k6/mixed.js — 90% fast requests (5 ms), 10% slow ones (200 ms)
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    open_model: { executor: 'constant-arrival-rate', rate: 100, timeUnit: '1s',
                  duration: '45s', preAllocatedVUs: 100, maxVUs: 1000 },
  },
};

export default function () {
  const ms = Math.random() < 0.1 ? 200 : 5;
  const res = http.get(`http://localhost:8080/api/cpu?ms=${ms}`, { tags: { kind: ms === 5 ? 'fast' : 'slow' } });
  check(res, { ok: (r) => r.status === 200 });
}

The average service time is 0.9 × 5 + 0.1 × 200 = 24.5 ms, so 100 req/s across 4 replicas is ≈ 61% utilization each.

2. Baseline: round robin (Nginx's default)

docker compose up -d --build --scale app=4
k6 run --summary-trend-stats="med,p(95),p(99)" k6/mixed.js

3. Switch to least connections

Add least_conn; as the first line inside upstream app_servers in nginx/nginx.conf, then reload Nginx without dropping connections:

docker compose exec nginx nginx -t && docker compose exec nginx nginx -s reload
k6 run --summary-trend-stats="med,p(95),p(99)" k6/mixed.js

Compare http_req_duration{kind:fast} between the runs — add thresholds: { 'http_req_duration{kind:fast}': ['p(99)<1000'] } to the options so k6 reports the tagged series. With round robin, fast requests frequently get stuck behind a 200 ms request on the same replica, so their p99 balloons. Least connections steers new work away from replicas busy with slow requests, cutting the fast requests' tail substantially.

4. Kill a replica mid-test

# in a second terminal while k6 runs
docker compose ps -q app | head -1 | xargs docker kill

Watch the error count. A kill (SIGKILL) gives the instance no chance to drain, so its in-flight requests fail; Nginx then marks it down via max_fails and retries idempotent GETs on other servers via proxy_next_upstream. Now repeat with docker stop (SIGTERM) on a replica running the graceful-shutdown code from 2.5 — the error count should be zero.

2.10

Key Takeaways

Next up, Module 03 — Caching at Every Layer: now that traffic is spread evenly, we make most of it disappear before it ever reaches a server.

Checkpoint

Module 02 Quiz

Covers layers, algorithms, hashing, health checks, draining, proxy configuration and global traffic.

3.1

Why Caching Works (and the Math Behind It)

A cache is a smaller, faster store that keeps copies of data whose original lives somewhere slower or more expensive. Caching is the single most effective scaling technique there is, because it doesn't make work faster — it makes work disappear. A request answered from a cache never touches your database at all.

Why it works: locality and skew

Caches work because access patterns are wildly uneven. Temporal locality: something used recently will probably be used again soon (today's homepage, a logged-in user's profile). Skew: a small fraction of items gets most of the traffic. Real popularity often follows a Zipf-like distribution, where the item ranked k is requested roughly in proportion to 1/k. The consequence is remarkable: a cache holding a tiny slice of the data can serve most requests.

The two numbers that matter

A cache's value comes down to its hit ratio h (fraction of lookups found in cache) and the latency gap between a hit and a miss:

Note the counterintuitive bit: going from 90% to 99% hit ratio looks like a 10% improvement, but it cuts the misses — the load on your database — by 10×, from 10% of traffic to 1%. At high hit ratios, every extra "nine" is worth as much as all the previous ones combined.

3.2

The Cache Hierarchy: From Browser to Disk

There isn't one cache in a web system — there's a chain of them, each closer to the user and smaller than the next. A request is answered by the first layer that has a fresh copy. The best-scaling systems push as much traffic as possible to the left of this diagram.

3.3

HTTP Caching: Cache-Control, Validators & Revalidation

The browser and CDN layers are controlled entirely by HTTP response headers. Getting these headers right is often the cheapest scaling win available: a few lines of configuration can remove most requests for static content from your servers entirely.

The Cache-Control directives you need

Validators: ETag and conditional requests

When a cached response expires, the cache doesn't have to download it again. If the response carried a validator — an ETag (a fingerprint of the content) or Last-Modified date — the cache asks "has it changed?" with If-None-Match or If-Modified-Since. If not, the server answers 304 Not Modified with no body. The round trip still happens, but the payload doesn't.

The golden pattern: fingerprinted assets

Static assets face a dilemma: long cache lifetimes are great for speed, but then how do users get the new version after a deploy? The answer used by virtually every modern build tool is to put a content hash in the filename: app.3f9a2c.js. The file at that URL can then never change, so it gets a one-year lifetime. The HTML page that references it is revalidated on every visit; when you deploy, the HTML points to app.8b1d07.js, a URL nobody has cached yet.

// Express: long-lived fingerprinted assets, always-fresh HTML
app.use('/assets', express.static('dist/assets', {
  maxAge: '1y',
  immutable: true,            // Cache-Control: public, max-age=31536000, immutable
}));

app.get('/', (req, res) => {
  res.set('Cache-Control', 'no-cache');  // store, but revalidate (ETag) every time
  res.sendFile(path.join(__dirname, 'dist/index.html'));
});

// A public, cacheable API response: browsers 30 s, CDN 5 min, serve stale while refreshing
app.get('/api/catalog', async (req, res) => {
  res.set('Cache-Control', 'public, max-age=30, s-maxage=300, stale-while-revalidate=60');
  res.json(await getCatalog());   // Express adds a weak ETag to JSON responses automatically
});

3.4

CDNs & Edge Caching

A Content Delivery Network is a globally distributed network of cache servers grouped into points of presence (PoPs), typically in hundreds of cities. Users connect to the nearest PoP (via anycast or DNS steering, as in Module 02). On a hit, the response comes from a few milliseconds away. On a miss, the PoP fetches from your origin, stores the response, and serves later requests itself.

What a CDN gives you beyond caching

• Shorter handshakes. TCP and TLS are set up with a nearby PoP, which then reuses warm, long-lived connections to your origin. Even uncacheable requests get faster.
• Origin offload and absorption. Bursts of traffic hit the CDN's huge capacity, not your servers — including much of a DDoS attack.
• Request collapsing. When 1,000 users request the same expired object at once, a good CDN sends one request to the origin and fans the response out.
• Edge logic. Redirects, header rewrites, A/B bucketing, auth checks and even full rendering can run at the edge (Cloudflare Workers, CloudFront Functions, Fastly Compute).

Tiered caching and the origin shield

With 300 PoPs, a newly published object could cause 300 separate origin fetches — one per PoP. An origin shield (tiered cache) designates a mid-tier cache in front of the origin: edge PoPs miss to the shield, and only the shield misses to the origin. Origin traffic drops sharply and hit ratios rise because the shield's cache is shared by all edges.

The cache key

The cache key decides which requests count as "the same". By default it's roughly scheme + host + path + query string. Tuning it is one of the most impactful CDN tasks: strip tracking parameters (?utm_source=…, ?fbclid=…) that make every URL unique, normalize query parameter order, and include only the headers that genuinely change the response (e.g. a device-class header instead of the raw User-Agent).

Purging

When content must change before its TTL ends, you purge it. Purging by URL works for single objects; for "everything about product 42" (its page, API responses, images), use surrogate keys / cache tags: the origin labels responses with tags like product-42, and one purge call invalidates every response carrying that tag across all PoPs, typically within seconds.

3.5

Application Caching Patterns

For data your application reads from a database, there are a handful of standard patterns describing who talks to the cache and the database, and when.

Why delete instead of update on writes?

In cache-aside, after writing to the database the app usually deletes the cache key rather than writing the new value into it. Deleting is idempotent and simple; the next reader repopulates the key from the database, which is the source of truth. Updating the cache directly invites a race: two concurrent writers can apply their DB writes in one order and their cache writes in the other, leaving the cache permanently wrong (until TTL).

3.6

Eviction, TTLs & Sizing

A cache is smaller than the data behind it, so it must constantly decide what to throw away. Two mechanisms do this: expiry (time-based) and eviction (space-based).

Eviction policies

• LRU (least recently used) — evict the item untouched for longest. The default almost everywhere; great for temporal locality.
• LFU (least frequently used) — evict the item with the fewest accesses. Better when popularity is stable; protects hot items from being flushed by one-off scans.
• FIFO / random — cheap and surprisingly decent at scale, but ignore access patterns.
• Modern hybrids — W-TinyLFU (used by Java's Caffeine) and similar algorithms combine recency and frequency with tiny memory overhead and beat pure LRU on most real workloads.

Choosing TTLs

A TTL bounds how stale data can be: a 60-second TTL guarantees no reader sees data more than ~60 seconds old, even if every explicit invalidation fails. Pick TTLs per data type by asking "how wrong can this be, for how long, before it hurts?" A product description can be minutes old; an account balance or stock count at checkout cannot be stale at all (read it from the source of truth). Even with perfect invalidation, keep a TTL as a safety net.

Sizing: the working set

The quantity to fit in cache is the working set — the data actively accessed within a time window — not the whole dataset. Estimate it as unique keys accessed per TTL window × average value size, then validate with real metrics: watch hit ratio and eviction rate as you change cache size. A cache that is evicting constantly with a mediocre hit ratio is too small; one with near-zero evictions and a flat hit ratio may be larger than needed. Remember the Zipf curve: doubling an already-large cache often buys only a few points of hit ratio.

3.7

Invalidation & Consistency

Phil Karlton's famous quip — "There are only two hard things in Computer Science: cache invalidation and naming things" — is famous because it's true. The moment you copy data, you have two versions that can disagree. Caching always trades some consistency for speed; the engineering is in controlling how much and for how long.

The classic stale-set race

Even the "delete on write" approach has a race. Follow the numbered steps:

Defenses, from simple to robust

• Always set a TTL. It bounds how long any inconsistency can last. The cheapest and most important defense.
• Delayed double delete. Delete the key on write, then delete it again a second or two later to catch any racing repopulation. Crude but effective.
• Leases / versioned writes. On a miss, the cache hands the reader a lease token; a later DEL invalidates the lease, and the reader's stale SET is rejected. Facebook's Memcache paper popularized this; you can approximate it with a version number in the value and a compare-and-set.
• Versioned keys. Include a version in the key (product:42:v7). Writers bump the version stored with the entity; readers build the key from it. Old entries simply age out.
• Change-data-capture (CDC) invalidation. Instead of app code deleting keys, a pipeline tails the database's replication log (e.g. Debezium on PostgreSQL/MySQL) and invalidates caches for every committed change, in commit order. Nothing is forgotten, even writes made by scripts or other services.

3.8

When Caches Hurt: Stampedes, Hot Keys, Penetration & Avalanches

Once a cache absorbs 95% of your traffic, your database is sized for the remaining 5%. That's the point — and also the danger. Anything that suddenly lowers the hit ratio multiplies the load on the database. These are the failure modes to design against.

Probabilistic early expiration

An elegant stampede defense: instead of every reader treating a key as valid until exactly its expiry, each reader independently decides to refresh slightly early with a probability that rises as expiry approaches. One lucky request recomputes the value before it expires, while everyone else keeps getting hits. This is known as "XFetch" from a 2015 paper by Vattani, Chierichetti and Lowenstein.

3.9

Distributed Caches: Redis & Memcached

An in-process cache is fastest, but with 50 app instances you get 50 separate caches, 50 cold starts and 50× the misses. A shared, networked cache fixes all three.

How Redis Cluster shards data

Redis Cluster divides the key space into 16,384 hash slots: slot = CRC16(key) mod 16384. Each primary node owns a range of slots, and each primary has replicas for failover. Adding a node means moving some slots to it — conceptually the same idea as consistent hashing: only a fraction of keys move. Multi-key commands only work when all keys live in the same slot; hash tags force that: {user:42}:profile and {user:42}:cart hash only the part in braces, so they land together.

3.10

Lab: Measure the Cache, Then Harden It

1. Add an uncached twin endpoint

Add to app/server.js so we can compare the same read with and without caching:

app.get('/api/products-nocache/:id', async (req, res, next) => {
  try {
    const { rows } = await db.query(
      // Same query as the cached route — but every request pays the DB round trip
      'SELECT id, name, price_cents FROM products WHERE id = $1',
      [req.params.id]
    );
    if (!rows[0]) return res.status(404).json({ error: 'not found' });
    res.json({ source: 'db', instance: INSTANCE, product: rows[0] });
  } catch (err) { next(err); }
});

2. A skewed workload

// k6/cache.js — 80% of requests target 20 hot products, 20% spread over 10,000
import http from 'k6/http';
const BASE = __ENV.BASE_URL || 'http://localhost:8080';
const PATH = __ENV.CACHED === '0' ? '/api/products-nocache/' : '/api/products/';

export const options = {
  scenarios: { load: { executor: 'constant-arrival-rate', rate: 500, timeUnit: '1s',
                       duration: '30s', preAllocatedVUs: 100, maxVUs: 1000 } },
};

export default function () {
  const id = Math.random() < 0.8 ? 1 + Math.floor(Math.random() * 20)
                                 : 1 + Math.floor(Math.random() * 10000);
  http.get(`${BASE}${PATH}${id}`);
}

docker compose up -d --build --scale app=3
k6 run -e CACHED=0 k6/cache.js      # every request hits PostgreSQL
k6 run -e CACHED=1 k6/cache.js      # cache-aside via Redis
docker compose exec redis redis-cli info stats | grep -E 'keyspace_(hits|misses)'

Compute the hit ratio from keyspace_hits / (keyspace_hits + keyspace_misses), and compare p95 latency and Postgres CPU (docker stats) between the runs.

3. Harden: coalescing + jittered TTLs + negative caching

// Request coalescing: concurrent misses for the same key share ONE database query
const inflight = new Map();
function singleflight(key, loader) {
  if (inflight.has(key)) return inflight.get(key);
  const p = loader().finally(() => inflight.delete(key));
  inflight.set(key, p);
  return p;
}

const ttl = (base) => base + Math.floor(Math.random() * base * 0.2); // +0–20% jitter
const NOT_FOUND = '__nf__';

app.get('/api/products/:id', async (req, res, next) => {
  try {
    const key = `product:${req.params.id}`;
    let cached = null;
    try { cached = await cache.get(key); } catch { /* cache down → fall through to DB */ }

    if (cached === NOT_FOUND) return res.status(404).json({ error: 'not found' });
    if (cached) return res.json({ source: 'cache', instance: INSTANCE, product: JSON.parse(cached) });

    const product = await singleflight(key, async () => {
      const { rows } = await db.query('SELECT id, name, price_cents FROM products WHERE id = $1', [req.params.id]);
      const value = rows[0] ? JSON.stringify(rows[0]) : NOT_FOUND;
      const expiry = rows[0] ? ttl(60) : 10;            // cache misses briefly
      cache.set(key, value, { EX: expiry }).catch(() => {});
      return rows[0] || null;
    });

    if (!product) return res.status(404).json({ error: 'not found' });
    res.json({ source: 'db', instance: INSTANCE, product });
  } catch (err) { next(err); }
});

4. Break it on purpose

docker compose stop redis      # while k6 is running
docker compose start redis

For the fallback to work, Redis calls must fail fast when Redis is unreachable. By default node-redis queues commands while disconnected and waits for a reconnect, so requests would hang instead. Create the client with createClient({ url: process.env.REDIS_URL, disableOfflineQueue: true }). Then the try/catch around cache.get lets the app fall back to the database instead of erroring. Watch latency and Postgres CPU climb while Redis is down — that's your system's "cache outage" capacity, a number worth knowing in production.

3.11

Key Takeaways

Next up, Module 04 — Scaling the Database: the 1–5% of traffic the caches can't absorb still has to land somewhere, and that somewhere holds all your state.

Checkpoint

Module 03 Quiz

Covers hit-ratio math, HTTP headers, CDNs, patterns, eviction, invalidation and failure modes.

4.1

Why the Database Is the Hard Part

Web servers are easy to scale because they're stateless: clone them and put a balancer in front. Caches are easy-ish because losing them only costs performance. The database is different. It holds the state — orders, balances, accounts — and it promises things the other tiers don't: that a committed write survives a crash (durability), that concurrent transactions don't corrupt each other (isolation), and that everyone agrees on the current value (consistency). Every way of spreading a database across machines puts at least one of those promises under strain.

The scaling ladder

Database scaling techniques differ enormously in cost and complexity. Climb this ladder in order, and only take the next step when measurements say the current one is exhausted. Teams that skip straight to sharding often discover that a missing index was the real problem.

1. 1Fix queries & add indexes. Usually the biggest win, often 10–1000× for the affected queries. Nearly free. (4.2)
2. 2Manage connections. Right-sized pools and a pooler like PgBouncer. (4.3)
3. 3Cache reads. Module 03 — remove most read traffic before it arrives.
4. 4Scale up. More RAM (fit the working set), faster storage, more cores. (4.4)
5. 5Read replicas. Spread reads across copies; accept replication lag. (4.5)
6. 6Partition & move data out. Split huge tables, archive cold data, move some workloads to specialized stores. (4.6, 4.8)
7. 7Shard writes across machines. The most powerful and most expensive step. (4.6)

4.2

Indexes, Query Plans & the N+1 Problem

Without an index, finding rows that match a condition means reading every row — a sequential scan, O(n). An index is a separate, sorted data structure that maps column values to row locations, so the database can jump straight to the matching rows. The default index type in PostgreSQL, MySQL and most relational databases is the B-tree.

Reading a query plan

Never guess why a query is slow; ask the database. EXPLAIN ANALYZE runs the query and shows the plan the optimizer chose, with real timings:

EXPLAIN ANALYZE SELECT * FROM products WHERE name = 'Product #77777';

 Seq Scan on products  (cost=0.00..1887.00 rows=1 width=22) (actual time=8.51..15.20 rows=1 loops=1)
   Filter: (name = 'Product #77777'::text)
   Rows Removed by Filter: 99999          -- read 100,000 rows to find 1
 Execution Time: 15.23 ms

CREATE INDEX idx_products_name ON products (name);

 Index Scan using idx_products_name on products  (cost=0.42..8.44 rows=1 width=22) (actual time=0.03..0.03 rows=1 loops=1)
   Index Cond: (name = 'Product #77777'::text)
 Execution Time: 0.05 ms                 -- ~300× faster

Signals to look for: Seq Scan on a big table with a selective filter, a large Rows Removed by Filter, estimated rows wildly different from actual rows (stale statistics — run ANALYZE), and sorts that spill to disk.

Index design rules

• Composite indexes follow the leftmost-prefix rule. An index on (tenant_id, created_at) serves WHERE tenant_id = ? and WHERE tenant_id = ? AND created_at > ?, but not WHERE created_at > ? alone. Put equality columns first, then range/sort columns.
• Covering indexes include every column the query needs (PostgreSQL: INCLUDE (...)), allowing an index-only scan that never touches the table.
• Indexes aren't free. Every insert/update must maintain each index, they consume RAM and disk, and unused ones are pure cost. Audit with pg_stat_user_indexes.
• Functions defeat indexes. WHERE lower(email) = ? can't use an index on email; create an expression index on lower(email) instead.
• Build indexes without downtime. On a live table, use CREATE INDEX CONCURRENTLY (PostgreSQL) so writes aren't blocked during the build.

The N+1 query problem

The most common performance bug in applications that use an ORM: one query to fetch a list, then one more query per item.

// ❌ N+1: 1 query for orders + 1 query per order. 100 orders = 101 round trips.
const orders = await db.query('SELECT id, product_id FROM orders WHERE user_id = $1', [userId]);
for (const o of orders.rows) {
  o.product = (await db.query('SELECT name FROM products WHERE id = $1', [o.product_id])).rows[0];
}

// ✅ 1 query with a JOIN (or 2 queries using WHERE id = ANY($1) for batching)
const { rows } = await db.query(`
  SELECT o.id, o.product_id, p.name AS product_name
  FROM orders o JOIN products p ON p.id = o.product_id
  WHERE o.user_id = $1`, [userId]);

At 0.5 ms per round trip, 101 queries cost ≥50 ms of pure network time — before the database does any work. Batching loaders (like the DataLoader pattern popular in GraphQL servers) collect IDs requested in one tick and fetch them in a single query.

Pagination that scales

LIMIT 20 OFFSET 100000 looks innocent, but the database must read and discard 100,000 rows to return 20 — page 5,000 is thousands of times slower than page 1. Keyset (cursor) pagination remembers where the last page ended and seeks directly there using the index:

-- ❌ Offset: cost grows with page number
SELECT id, name FROM products ORDER BY id LIMIT 20 OFFSET 100000;

-- ✅ Keyset: constant cost on every page (the client sends the last id it saw)
SELECT id, name FROM products WHERE id > 100000 ORDER BY id LIMIT 20;

4.3

Connection Management: Pools & PgBouncer

Database connections are expensive. PostgreSQL forks a dedicated operating-system process per connection, each using several megabytes of memory plus per-connection bookkeeping, and performance degrades as hundreds of them contend for locks and CPU. Its default max_connections is just 100. MySQL uses threads rather than processes, but the same principle holds: a database does its best work with a modest number of busy connections, not thousands of idle ones.

Pool modes

• Session pooling — a client keeps one server connection until it disconnects. Safe for every feature, but saves little.
• Transaction pooling — a server connection is assigned only for the duration of a transaction, then returned. This is where the huge multiplexing gains come from. The catch: session-level state doesn't persist between transactions — SET variables, session advisory locks, LISTEN, temporary tables. Prepared statements historically broke too; PgBouncer 1.21+ supports protocol-level prepared statements when configured (max_prepared_statements).
• Statement pooling — returned after every statement; multi-statement transactions are disallowed. Rarely used.

Sizing the pool

Use Little's Law from Module 01: busy connections = query rate × time each query holds a connection. 2,000 queries/s × 5 ms = 10 busy connections on average. A database server rarely benefits from more actively running connections than a small multiple of its CPU cores (plus some for I/O waits). A pool that's too large doesn't add throughput; it adds contention and moves the queue inside the database, where it's harder to see. Bound total connections and let requests queue briefly in the app's pool, with a short acquire timeout so overload fails fast.

const db = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10,                        // per instance; total = instances × max
  idleTimeoutMillis: 30_000,      // release idle connections
  connectionTimeoutMillis: 2_000, // fail fast if no connection is free in 2 s
  statement_timeout: 5_000,       // server-side cap on any single query
});

4.4

Scaling Up: RAM, Storage & Tuning

Before distributing a database, make the single node as capable as possible. Vertical scaling keeps every guarantee intact, and modern hardware goes very far: cloud instances with hundreds of vCPUs, terabytes of RAM and NVMe storage delivering hundreds of thousands of IOPS.

• Fit the working set in RAM. Databases cache hot table and index pages in memory (PostgreSQL's shared_buffers plus the OS page cache; InnoDB's buffer pool). When the working set fits, most reads never touch disk. When it outgrows RAM, latency jumps suddenly as reads hit storage. Track the buffer-cache hit ratio; for OLTP workloads it should stay very high (often above 99%).
• Storage IOPS and latency. Writes must reach durable storage (the write-ahead log is flushed on commit). Provisioned-IOPS volumes or local NVMe can be the difference between 2 ms and 20 ms commits.
• Keep tables healthy. PostgreSQL's MVCC leaves dead row versions behind after updates and deletes; autovacuum reclaims them. A table whose vacuum can't keep up bloats, gets slower, and in extreme cases risks transaction-ID wraparound. Tune autovacuum for hot tables rather than disabling it.
• Find the expensive queries. pg_stat_statements ranks queries by total time. Optimizing the top five usually beats any hardware upgrade.

4.5

Replication, Read Replicas & Lag

Replication keeps copies of the same data on several machines. The most common form is leader-follower (primary-replica): all writes go to the primary, which streams its change log — PostgreSQL's write-ahead log (WAL), MySQL's binlog — to replicas that replay it. Replication serves three goals: read scaling (replicas serve reads), high availability (promote a replica if the primary dies) and geographic locality (replicas near users).

Synchronous vs asynchronous

With synchronous replication, the primary reports a commit only after a replica confirms receiving it — no acknowledged write can be lost if the primary dies, but every commit pays a network round trip, and if the synchronous replica becomes unavailable, writes stall. With asynchronous replication, the primary commits immediately; replicas catch up "soon". Most production setups use a hybrid: one synchronous standby (often in another availability zone) for durability, plus asynchronous replicas for read scaling.

Replication lag and its anomalies

Asynchronous replicas are usually milliseconds behind, but under heavy write load, long-running queries or network trouble, lag can grow to seconds or minutes. Two user-visible anomalies follow:

• Read-your-writes violation — a user updates their profile (primary), the next page load reads a replica that hasn't caught up, and the change seems to have vanished.
• Non-monotonic reads — successive reads hit different replicas with different lag, so data appears to go backwards in time (a comment appears, then disappears on refresh).

const primary = new Pool({ connectionString: process.env.DATABASE_URL, max: 10 });
const replica = new Pool({ connectionString: process.env.REPLICA_URL, max: 10 });

// Read-your-writes: for a few seconds after a user writes, send their reads to the primary.
function poolFor(req) {
  const lastWrite = Number(req.cookies?.lastWriteAt || 0);   // set by write handlers
  return Date.now() - lastWrite < 5_000 ? primary : replica;
}

app.get('/api/me', async (req, res) => {
  const { rows } = await poolFor(req).query('SELECT * FROM users WHERE id = $1', [req.user.id]);
  res.json(rows[0]);
});

For monotonic reads, pin each user to one replica (e.g. hash of user ID) rather than choosing randomly per request.

Failover

When the primary fails, a replica must be promoted, and clients must be redirected to it. Tools like Patroni (PostgreSQL) or managed services (Amazon RDS/Aurora, Cloud SQL, Azure Database) automate detection and promotion, typically in tens of seconds. Two dangers: with asynchronous replication, writes not yet replicated are lost on promotion; and if the old primary isn't truly dead (just partitioned), you can get split brain — two primaries accepting conflicting writes. Consensus-based leader election and fencing (forcibly cutting off the old primary) exist to prevent this.

4.6

Partitioning & Sharding

When a single primary can no longer handle the write volume or hold the data, you must split the data itself. Partitioning splits a large table into smaller pieces, usually within one database server. Sharding (horizontal partitioning across machines) places different subsets of rows on different database servers, each acting as primary for its own slice. Each shard is smaller, faster, and handles a fraction of the writes.

Partitioning within one server

PostgreSQL's declarative partitioning splits a table by range, list or hash. Its killer use case is time-series data: partition events by month, and queries for "last 7 days" only touch one or two partitions (partition pruning). Retention becomes instant — DROP last year's partition instead of a slow, bloat-inducing DELETE of millions of rows.

CREATE TABLE events (
  id         BIGINT GENERATED ALWAYS AS IDENTITY,
  user_id    BIGINT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL,
  payload    JSONB
) PARTITION BY RANGE (created_at);

CREATE TABLE events_2026_10 PARTITION OF events
  FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');

Choosing a shard key

Everything about sharding depends on the shard key: the column that decides which shard owns a row. A good shard key has high cardinality, spreads load (not just data) evenly, and matches your access patterns so most queries touch a single shard. For a SaaS product that's often tenant_id; for a consumer app, user_id.

Interactive: how evenly does your shard key spread writes?

The same 20,000 writes spread over 8 shards using different shard keys. The ideal is 12.5% per shard.

The costs of sharding

• Cross-shard queries. "Top 10 products across all users" must scatter to every shard and gather the results. The slowest shard sets the latency (the tail-at-scale problem from Module 01).
• Cross-shard transactions. Moving money between users on different shards needs a distributed protocol (two-phase commit) or a redesign around sagas (Module 05). Both are complex.
• Global uniqueness. Auto-increment IDs collide across shards. Use UUIDs (preferably time-ordered UUIDv7 for index locality) or Snowflake-style IDs (timestamp + machine + sequence).
• Resharding. Growing from 8 to 16 shards means moving live data. Designs that pre-split into many logical shards (e.g. 1,024) mapped onto fewer physical servers make this a matter of moving whole logical shards.
• Operational load. Backups, schema migrations, monitoring and failover — multiplied by the number of shards.

4.7

Consistency Models, Quorums & Isolation Levels

"Consistency" means several different things in databases. Two families matter for scaling: consistency models (what readers see across replicas) and isolation levels (how concurrent transactions interact on one database).

Consistency models, strongest to weakest

Quorums

Leaderless databases (Cassandra, DynamoDB-style systems) write to N replicas and let you tune how many must respond: W for writes, R for reads. If R + W > N, every read set overlaps every write set in at least one replica, so a read sees the latest acknowledged write (with caveats around concurrent writes and failures). With N = 3, W = 2, R = 2 you tolerate one slow or failed node on both paths. Lowering R or W to 1 buys latency and availability by giving up that overlap — PACELC in action.

Isolation levels and anomalies

Even on one server, concurrent transactions can interfere. Isolation levels define which anomalies are prevented:

• Read Committed (PostgreSQL's default) — you never see uncommitted data, but two reads in one transaction can see different values.
• Repeatable Read / Snapshot Isolation (MySQL InnoDB's default level is Repeatable Read) — the transaction sees a consistent snapshot. Still allows write skew: two doctors each check "at least one other doctor is on call" and both go off call.
• Serializable — results are as if transactions ran one at a time. PostgreSQL implements it efficiently (SSI) but aborts conflicting transactions, so apps must retry.

The most common real-world bug is the lost update (read-modify-write in application code). Prefer atomic statements or explicit locks:

-- ❌ read stock in app code, compute stock - 1, write it back: concurrent buyers overwrite each other

-- ✅ atomic and safe under concurrency; returns no row if sold out
UPDATE products SET stock = stock - 1
WHERE id = $1 AND stock > 0
RETURNING stock;

-- ✅ or lock the row for the rest of the transaction
SELECT stock FROM products WHERE id = $1 FOR UPDATE;

4.8

Choosing a Data Store: SQL, NoSQL & NewSQL

"NoSQL scales, SQL doesn't" was a slogan from the early 2010s, and it was always too simple. What scales is a data model that matches the access pattern and a partitioning scheme that keeps most operations on one node. Many NoSQL databases scale out easily because they make you design for that up front, and give up features (joins, multi-row transactions) that are hard to distribute.

Large systems are usually polyglot: a relational database as the source of truth, Redis for caching and counters, a search engine fed from the database for text queries, object storage for files, and a warehouse for analytics. The data flows between them via change-data-capture and events — the subject of Module 05.

4.9

Lab: Indexes, Pagination & Connection Exhaustion

1. Feel a missing index

docker compose exec postgres psql -U lab -d lab

\timing on
EXPLAIN ANALYZE SELECT * FROM products WHERE name = 'Product #77777';
CREATE INDEX CONCURRENTLY idx_products_name ON products (name);
EXPLAIN ANALYZE SELECT * FROM products WHERE name = 'Product #77777';

-- pagination: compare the plans and timings
EXPLAIN ANALYZE SELECT id, name FROM products ORDER BY id LIMIT 20 OFFSET 90000;
EXPLAIN ANALYZE SELECT id, name FROM products WHERE id > 90000 ORDER BY id LIMIT 20;

Note the "Rows Removed by Filter" before the index, and how the OFFSET plan reads ~90,020 rows while the keyset plan reads 20.

2. Exhaust the connection limit

Each app replica has a pool of up to 10 connections, and PostgreSQL's default max_connections is 100 (a few are reserved for superusers). Fast queries release connections so quickly that pools rarely fill, so add an endpoint that holds a connection for 200 ms:

// app/server.js — simulates a slow query (a report, a lock wait, a missing index)
app.get('/api/slow-query', async (req, res, next) => {
  try {
    await db.query('SELECT pg_sleep(0.2)');
    res.json({ ok: true, instance: INSTANCE });
  } catch (err) { next(err); }
});

// k6/slowdb.js — 600 req/s × 0.2 s = 120 connections needed (Little's Law)
import http from 'k6/http';
import { check } from 'k6';
export const options = {
  scenarios: { load: { executor: 'constant-arrival-rate', rate: 600, timeUnit: '1s',
                       duration: '30s', preAllocatedVUs: 200, maxVUs: 2000 } },
};
export default function () {
  check(http.get('http://localhost:8080/api/slow-query'), { ok: (r) => r.status === 200 });
}

docker compose up -d --build --scale app=12   # up to 12 × 10 = 120 connections
k6 run k6/slowdb.js
docker compose logs app | grep -i "too many clients"

You should see errors such as sorry, too many clients already and a non-zero http_req_failed rate. The scale-out added failures, not capacity.

3. Fix it with PgBouncer

  pgbouncer:
    image: edoburu/pgbouncer:latest     # pin a specific tag in real projects
    environment:
      DB_HOST: postgres
      DB_USER: lab
      DB_PASSWORD: lab
      DB_NAME: lab
      AUTH_TYPE: scram-sha-256          # matches PostgreSQL 16's default
      POOL_MODE: transaction
      MAX_CLIENT_CONN: 2000
      DEFAULT_POOL_SIZE: 30             # real connections to PostgreSQL
    depends_on: [postgres]

  app:
    environment:
      DATABASE_URL: postgres://lab:lab@pgbouncer:5432/lab   # point the app at the pooler

Re-run the test with 12 replicas. The "too many clients" errors disappear, because PgBouncer multiplexes up to 120 client connections onto 30 server connections. Latency rises, though: 30 connections × 5 queries/s each = 150 queries/s, far below the 600 you are sending, so requests now wait in PgBouncer instead of failing. That's the honest result. A pooler protects the database from connection storms and turns hard failures into queueing, but it can't create capacity. Here, only a faster query would (or caching, or fewer slow requests). Check the image's README if the environment variable names differ in newer versions.

4.10

Key Takeaways

Next up, Module 05 — Asynchronous Processing: moving slow work off the request path and connecting services and data stores with queues and event streams.

Checkpoint

Module 04 Quiz

Covers indexes, pooling, replication, sharding, consistency and data-store choice.

5.1

Why Go Asynchronous?

So far, every request has been synchronous: the user waits while the server does everything. That's fine when "everything" is a 5 ms query. It's a problem when placing an order also means charging a card, sending an email, generating a PDF invoice, updating search indexes, notifying the warehouse and recording analytics. Every one of those steps adds latency, and every one is a dependency that can fail and take the order down with it.

What asynchronous processing buys you

• Lower latency — users wait only for the critical path.
• Load leveling — a spike of 10,000 orders in a minute becomes a queue that workers drain at a steady, sustainable rate instead of overwhelming downstream systems.
• Fault isolation — if the email provider is down for 10 minutes, emails wait in the queue and are retried; orders keep flowing.
• Independent scaling — web servers scale on request rate; workers scale on queue depth; each tier sized for its own job.
• Decoupling — the order service doesn't need to know who cares about new orders. New consumers (fraud detection, recommendations) subscribe without touching it.

5.2

Three Messaging Models: Queues, Pub/Sub & Logs

5.3

Delivery Semantics & Idempotent Consumers

Networks fail and processes crash at the worst possible moment — after a consumer finished the work but before it acknowledged the message, for instance. Messaging systems therefore offer one of three delivery guarantees:

• At-most-once — acknowledge first, then process. A crash loses the message. Acceptable for metrics or logs where a small loss doesn't matter.
• At-least-once — process first, then acknowledge. A crash before the ack means the message is redelivered, so it may be processed twice. The default for almost every system.
• Exactly-once — every message affects the system exactly once. True exactly-once delivery across arbitrary systems is impossible in the presence of failures; what systems actually provide is effectively-once processing: at-least-once delivery plus deduplication or idempotent effects.

Visibility timeouts and acknowledgements

In SQS, receiving a message hides it from other consumers for a visibility timeout (default 30 seconds). If the consumer deletes it in time, it's done; if not — crash, or processing simply took too long — it reappears and another consumer receives it. RabbitMQ does the same with unacknowledged deliveries when a consumer's channel closes. If your processing time can exceed the timeout, extend it while working, or you'll process messages twice without any crash at all.

Making consumers idempotent

Since duplicates will happen, design every consumer so processing the same message twice has the same effect as once:

// Dedupe table: CREATE TABLE processed_messages (message_id TEXT PRIMARY KEY, processed_at TIMESTAMPTZ DEFAULT now());
async function handleOrderPaid(msg) {
  const client = await db.connect();
  try {
    await client.query('BEGIN');
    const { rowCount } = await client.query(
      'INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING', [msg.id]);
    if (rowCount === 0) { await client.query('ROLLBACK'); return; }   // seen before → skip

    await client.query('UPDATE orders SET status = $1 WHERE id = $2', ['paid', msg.orderId]);
    await client.query('INSERT INTO loyalty_points (user_id, points, order_id) VALUES ($1, $2, $3)',
      [msg.userId, msg.points, msg.orderId]);
    await client.query('COMMIT');   // dedupe record and effects commit atomically
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;                      // message will be redelivered and retried
  } finally {
    client.release();
  }
}

Other idempotency techniques: natural idempotence (SET status = 'paid' is safe to repeat; points = points + 10 is not), unique constraints (one loyalty row per order_id), conditional updates (WHERE status = 'pending'), and passing idempotency keys to external APIs.

5.4

Ordering, Partitions & Consumer Groups

With many consumers working in parallel, global ordering is gone: message 2 may finish before message 1. Often that's fine. When it isn't — "item added" must be applied before "item removed" for the same cart — you need ordering per entity. Partitioned logs like Kafka provide exactly this.

• Key → partition. Messages with the same key always go to the same partition, and a partition is an ordered log. So ordering is guaranteed per key, not globally.
• One consumer per partition per group. Within a consumer group, each partition is assigned to exactly one consumer. Parallelism is therefore capped at the partition count — plan partitions for future throughput, since increasing them later changes the key → partition mapping.
• Rebalancing. When consumers join or leave, partitions are reassigned. Processing pauses briefly, and messages in flight may be reprocessed — another reason for idempotency.
• Hot partitions. A skewed key (one giant tenant) overloads one partition and one consumer, exactly like a hot shard in Module 04.

5.5

Load Leveling, Back-Pressure, Retries & Dead Letters

A queue is a shock absorber. When producers temporarily outpace consumers, the backlog grows; when the burst ends, consumers drain it. The key numbers are the arrival rate, the drain rate (consumers × throughput each), and the backlog. By Little's Law, a message's wait ≈ backlog ÷ drain rate.

Interactive: a traffic spike hits a queue

Notice two things. First, the backlog keeps growing for as long as arrivals exceed capacity, and then takes far longer to drain than the spike lasted, because only the spare capacity drains it. Second, if steady-state arrivals exceed capacity, the backlog never drains — no queue fixes a system that's simply too small.

Back-pressure

Queues can't grow forever: memory and disk run out, and messages that wait an hour may be worthless (a "your code is 123456" SMS). Back-pressure means pushing the overload signal back toward the source: bounded queues that reject new work when full (the producer returns 503/429 or slows down), consumer prefetch limits so one consumer doesn't grab more than it can handle, and message TTLs that drop work too stale to matter.

Retries with exponential backoff and jitter

When processing fails because a dependency is down, retrying immediately just hammers it. Wait longer after each failure — exponential backoff — and add jitter (randomness) so thousands of failed messages don't all retry at the same instant:

// "Full jitter": sleep a random time between 0 and the exponential cap
function backoffMs(attempt, baseMs = 500, capMs = 60_000) {
  const exp = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.random() * exp;
}
// attempt 0 → 0–0.5 s, 1 → 0–1 s, 2 → 0–2 s, … 7+ → 0–60 s

Poison messages and dead-letter queues

Some messages will never succeed: malformed JSON, a reference to a deleted record, a bug triggered by unusual input. Retried forever, a poison message wastes capacity and, in ordered partitions, blocks every message behind it. After a maximum number of attempts, move it to a dead-letter queue (DLQ) for humans to inspect, fix and replay. Every queue in production should have a DLQ — and an alert when anything lands in it.

5.6

Reliable Publishing: the Dual-Write Problem & the Outbox Pattern

A deceptively simple piece of code hides one of the most common data-integrity bugs in distributed systems:

// ❌ Dual write: two systems, no shared transaction
await db.query('INSERT INTO orders ...');             // 1. commits
await broker.publish('order.created', { orderId });   // 2. what if this fails, or we crash here?

If the process crashes between the two lines, the order exists but no event was published: no email, no warehouse notification, ever. Reverse the order and you can publish an event for an order that then fails to commit. There's no transaction that spans a database and a message broker, so "just do both" can't be made reliable.

CREATE TABLE outbox (
  id           BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  topic        TEXT NOT NULL,
  payload      JSONB NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ
);

// 1. In the request: business row + event row, atomically
await client.query('BEGIN');
const { rows } = await client.query(
  'INSERT INTO orders (user_id, total_cents) VALUES ($1, $2) RETURNING id', [userId, total]);
await client.query('INSERT INTO outbox (topic, payload) VALUES ($1, $2)',
  ['order.created', { orderId: rows[0].id, userId }]);
await client.query('COMMIT');

// 2–3. Relay loop (a separate worker). SKIP LOCKED lets several relays run safely in parallel.
async function relayOnce() {
  const client = await db.connect();
  try {
    await client.query('BEGIN');
    const { rows } = await client.query(`
      SELECT id, topic, payload FROM outbox
      WHERE published_at IS NULL ORDER BY id LIMIT 100
      FOR UPDATE SKIP LOCKED`);
    for (const row of rows) await broker.publish(row.topic, row.payload, { messageId: String(row.id) });
    await client.query('UPDATE outbox SET published_at = now() WHERE id = ANY($1)', [rows.map((r) => r.id)]);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

Instead of polling, the relay can be a CDC connector (e.g. Debezium) tailing the database's replication log and publishing outbox inserts as they commit — lower latency and no polling load. The reverse problem (consume a message and update a database exactly once) is handled by the idempotent-consumer pattern from 5.3, sometimes called the inbox pattern.

5.7

Event-Driven Architecture & Sagas

Commands vs events

A command asks someone to do something (SendInvoice) — it has one intended handler and can be rejected. An event states a fact that already happened (OrderPlaced) — any number of consumers may react, and the publisher doesn't care who. Systems built around events are loosely coupled: you can add a fraud-detection consumer without changing the order service at all. The cost is that the end-to-end flow is no longer visible in one place in the code.

Sagas: transactions across services

When an operation spans several services with their own databases (reserve stock, charge payment, schedule shipping), there's no single ACID transaction. A saga breaks it into local transactions, each publishing an event or message that triggers the next. If a step fails, previous steps are undone by compensating actions (release the stock, refund the payment). Compensation is a business-level "undo", not a rollback — a refund is visible, an email can't be unsent.

5.8

Background Jobs & Scheduled Work

Most applications don't need Kafka on day one. They need a job queue: "run this function later, retry it if it fails". Libraries such as BullMQ (Node.js, on Redis), Sidekiq (Ruby), Celery (Python) and the many SQL-backed queues (e.g. using SELECT … FOR UPDATE SKIP LOCKED in PostgreSQL) cover this with retries, delays, priorities, rate limits and dashboards.

• Run workers as a separate deployment from the web tier, so a CPU-heavy job can't slow page loads and each tier scales on its own signal.
• Keep job payloads small: pass IDs, not whole objects. The worker loads fresh data, and you avoid bloating the queue.
• Make jobs idempotent and resumable: a job processing 1 million rows should checkpoint its progress, so a retry doesn't start from zero.
• Scheduled (cron) work in a scaled system: remember Module 01's "10 replicas send the email 10 times". Use a single scheduler that enqueues jobs (BullMQ repeatable jobs, Kubernetes CronJobs, a cloud scheduler), or leader election — never a timer in every web process.

5.9

Lab: Move Slow Work to a BullMQ Worker

We'll add an order endpoint whose confirmation email takes 300 ms, first synchronously, then via a queue and a separately scaled worker.

1. Schema and dependency

-- db/orders.sql
CREATE TABLE IF NOT EXISTS orders (
  id                   BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  product_id           INT NOT NULL,
  email                TEXT NOT NULL,
  created_at           TIMESTAMPTZ NOT NULL DEFAULT now(),
  confirmation_sent_at TIMESTAMPTZ
);

docker compose exec -T postgres psql -U lab -d lab < db/orders.sql
cd app && npm install bullmq && cd ..

2. Shared queue module and a fake email sender

// app/queue.js
const { Queue } = require('bullmq');
const connection = { host: 'redis', port: 6379 };
const emailQueue = new Queue('emails', { connection });

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function sendConfirmationEmail(orderId) {
  await sleep(300);                                   // pretend: a slow third-party API
  if (Math.random() < 0.1) throw new Error('email provider timeout');  // and a flaky one
}

module.exports = { connection, emailQueue, sendConfirmationEmail };

3. Two versions of the endpoint

// app/server.js
const { emailQueue, sendConfirmationEmail } = require('./queue');
app.use(express.json());

// ❌ Synchronous: the user waits 300 ms, and 10% of orders fail with a 500
app.post('/api/orders-sync', async (req, res, next) => {
  try {
    const { rows } = await db.query(
      'INSERT INTO orders (product_id, email) VALUES ($1, $2) RETURNING id', [req.body.productId, req.body.email]);
    await sendConfirmationEmail(rows[0].id);
    await db.query('UPDATE orders SET confirmation_sent_at = now() WHERE id = $1', [rows[0].id]);
    res.status(201).json({ orderId: rows[0].id });
  } catch (err) { next(err); }
});

// ✅ Asynchronous: enqueue and return immediately
app.post('/api/orders', async (req, res, next) => {
  try {
    const { rows } = await db.query(
      'INSERT INTO orders (product_id, email) VALUES ($1, $2) RETURNING id', [req.body.productId, req.body.email]);
    const orderId = rows[0].id;
    await emailQueue.add('confirm', { orderId }, {
      jobId: `confirm-${orderId}`,                    // idempotent enqueue: duplicates are ignored
      attempts: 5,
      backoff: { type: 'exponential', delay: 1000 },  // 1 s, 2 s, 4 s, 8 s…
      removeOnComplete: 1000,
      removeOnFail: false,                            // failed jobs stay inspectable (our DLQ)
    });
    res.status(202).json({ orderId, status: 'confirmation queued' });
  } catch (err) { next(err); }
});

4. The worker — a separate process

// app/worker.js
const { Worker } = require('bullmq');
const { Pool } = require('pg');
const { connection, sendConfirmationEmail } = require('./queue');

const db = new Pool({ connectionString: process.env.DATABASE_URL, max: 5 });

const worker = new Worker('emails', async (job) => {
  const { orderId } = job.data;
  // Idempotency: skip if a previous attempt already finished
  const { rows } = await db.query('SELECT confirmation_sent_at FROM orders WHERE id = $1', [orderId]);
  if (!rows[0] || rows[0].confirmation_sent_at) return 'skipped';

  await sendConfirmationEmail(orderId);       // throws → BullMQ retries with backoff
  await db.query('UPDATE orders SET confirmation_sent_at = now() WHERE id = $1', [orderId]);
  return 'sent';
}, { connection: { ...connection, maxRetriesPerRequest: null }, concurrency: 20 });

worker.on('failed', (job, err) => console.warn(`job ${job?.id} attempt ${job?.attemptsMade} failed: ${err.message}`));

process.on('SIGTERM', async () => { await worker.close(); await db.end(); process.exit(0); });  // graceful drain

# docker-compose.yml — the worker reuses the app image with a different command
  worker:
    build: ./app
    command: ["node", "worker.js"]
    environment:
      DATABASE_URL: postgres://lab:lab@postgres:5432/lab
    depends_on: [redis, postgres]

5. Compare

// k6/orders.js — run with -e EP=orders-sync, then -e EP=orders
import http from 'k6/http';
export const options = { scenarios: { load: { executor: 'constant-arrival-rate', rate: 50,
  timeUnit: '1s', duration: '30s', preAllocatedVUs: 50, maxVUs: 500 } } };
export default function () {
  http.post(`http://localhost:8080/api/${__ENV.EP}`,
    JSON.stringify({ productId: 42, email: 'test@example.com' }),
    { headers: { 'Content-Type': 'application/json' } });
}

docker compose up -d --build --scale app=3 --scale worker=2
k6 run -e EP=orders-sync k6/orders.js    # p95 > 300 ms and ~10% failed requests
k6 run -e EP=orders k6/orders.js         # p95 of a few ms, ~0% failed
docker compose logs -f worker            # watch retries succeed with backoff

With the queue, users never see the email provider's latency or its 10% failure rate: failed attempts retry in the background, and with 5 attempts the chance a job fails permanently is 0.1⁵ = 0.001%. Now stop the workers for 30 seconds (docker compose stop worker) during a run — orders keep succeeding, jobs pile up, and they drain as soon as the workers return. That's load leveling and fault isolation on your own laptop.

5.10

Key Takeaways

Next up, Module 06 — Service Architecture & APIs: monoliths, microservices, and designing the APIs that connect them at scale.

Checkpoint

Module 05 Quiz

Covers messaging models, delivery guarantees, idempotency, ordering, back-pressure, outbox and sagas.

6.1

Monoliths, Modular Monoliths & Microservices

Everything so far has scaled traffic. Successful products also have to scale people: when 200 engineers work in one codebase, deploys queue up, one team's bug blocks everyone's release, and nobody understands the whole system. Service architecture is mostly about how you split software so that both the system and the organization can grow.

Conway's Law

Melvin Conway observed in 1968 that organizations design systems that mirror their own communication structure. Service boundaries tend to follow team boundaries whether you plan it or not. Many companies now use this deliberately (the "inverse Conway maneuver"): shape teams around the architecture you want, and give each team end-to-end ownership of a service.

When to split

Microservices are an answer to organizational scale far more than traffic scale — a stateless monolith scales horizontally behind a load balancer just fine (Module 02). Good reasons to extract a service: a part of the system needs to deploy independently at a different pace, has very different scaling or resource needs (a GPU image pipeline, a memory-hungry search index), needs stronger isolation (payments, for compliance or blast radius), or is owned by a separate team. Draw boundaries around business capabilities (billing, catalog, shipping), not technical layers ("the database service").

6.2

The Cost of Distribution

Splitting a function call into a network call changes its nature. An in-process call takes nanoseconds and either returns or throws. A network call takes milliseconds, can time out with no idea whether the other side did the work, and fails in ways that depend on machines you don't control. The classic fallacies of distributed computing (from Peter Deutsch and colleagues at Sun) list the assumptions that bite newcomers: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn't change; there is one administrator; transport cost is zero; the network is homogeneous.

Latency and availability compound

Module 01's series-availability math applies directly to service call chains. A request that passes synchronously through five services, each 99.9% available, is at most 99.5% available — before counting the network between them. Latency adds up the same way, and tail latency gets worse with every hop and every fan-out.

Interactive: what does a call chain cost?

6.3

API Styles: REST, gRPC, GraphQL & Messaging

Schemas and contracts

Whatever the style, the API is a contract between teams that deploy independently. Describe it in a machine-readable schema (OpenAPI for REST, .proto files for gRPC, the GraphQL schema), generate clients from it, and check compatibility automatically in CI so a breaking change is caught before it ships.

6.4

Designing APIs That Scale

An API's design decides how much work each client action costs your servers — and how easily you can change things later. These habits pay off at every scale:

• Paginate every list with cursors (Module 04's keyset pagination), return a next_cursor, and cap page size. An unbounded GET /orders is an outage waiting for one big customer.
• Let clients ask for less: field selection (?fields=id,name), filters, and compact representations reduce bytes and work.
• Let clients ask once: batch endpoints (GET /products?ids=1,2,3) beat 3 round trips, especially from mobile networks.
• Make reads cacheable: stable URLs, Cache-Control, ETags and conditional requests (Module 03).
• Make writes retry-safe with idempotency keys (below).
• Publish rate limits and return 429 with Retry-After so well-behaved clients back off (Module 10).
• Use asynchronous APIs for long work: return 202 Accepted with a status URL instead of holding a request open for 2 minutes.

Idempotency keys

A mobile client sends POST /payments, the connection drops, and the client doesn't know whether the payment happened. If it retries, the customer may be charged twice. The standard fix, popularized by Stripe's API: the client generates a unique idempotency key per logical operation and sends it in a header. The server stores the result under that key; a retry with the same key gets the stored result instead of a second charge.

POST /api/payments HTTP/1.1
Idempotency-Key: 7f3c9a2e-1b4d-4f8e-9c2a-5d6e7f8a9b0c
Content-Type: application/json

{"orderId": 981, "amountCents": 4999}

Evolving APIs without breaking clients

With independent deploys and mobile apps that users never update, old and new versions of clients and servers always coexist. Rules that keep you safe:

• Additive changes only: adding optional fields and new endpoints is safe; renaming, removing, or changing the meaning or type of a field is breaking.
• Tolerant readers: clients ignore fields they don't know; servers accept requests without new optional fields.
• Expand → migrate → contract: add the new field alongside the old, move all clients, and only then remove the old one.
• Version explicitly when you must break (/v2/ path, header or media type), and run versions side by side with a published deprecation timeline.

6.5

API Gateways & Backends-for-Frontends

With dozens of services, you don't want every client to know every service's address, and you don't want every service to reimplement authentication and rate limiting. An API gateway is the single front door: it routes requests to services and handles cross-cutting concerns once.

A backend-for-frontend (BFF) is a thin service owned by a client team that aggregates internal calls into the exact payload one client needs: a phone on a slow network gets one compact response instead of making six calls. BFFs also absorb the churn of UI changes so core services' APIs stay stable.

6.6

Service Discovery & the Service Mesh

With autoscaling and rolling deploys, service instances appear and disappear constantly. Callers need to find healthy instances at runtime — service discovery.

• DNS-based: each service has a name that resolves to its instances or to a virtual IP (Kubernetes Services, Docker Compose). Simple and universal; watch out for DNS caching in clients.
• Registry-based: instances register themselves with a registry (Consul, etcd, ZooKeeper) and clients query it. Client-side discovery means the caller picks an instance itself (and load-balances, e.g. with power of two choices); server-side discovery means it calls a load balancer that does the lookup.

The service mesh

Every service needs the same networking features: timeouts, retries with budgets, load balancing, mutual TLS, metrics and tracing. Rather than implementing them in every language's libraries, a service mesh (Istio, Linkerd, Consul Connect) puts a small proxy — usually Envoy — next to each instance as a sidecar (or, in newer designs, a per-node proxy). All traffic flows through these proxies, which a central control plane configures.

The trade-off: an extra hop (sub-millisecond, but real), extra CPU and memory per instance, and a significant new piece of infrastructure to operate and debug. A mesh pays off with many services and polyglot teams; with five services and one language, a good shared HTTP client library is often enough.

6.7

Data Ownership, Composition & CQRS

The rule that makes microservices work: each service owns its data, and other services access it only through that service's API or events — never by querying its tables. A shared database couples services at the schema level: one team's migration breaks another team's queries, and no one can scale or change their storage independently.

But I need data from three services on one page…

• API composition: a BFF or composer calls the services in parallel and merges results. Simple; latency is set by the slowest call, and availability by the product of all of them.
• Local read models (materialized views): a service subscribes to other services' events and keeps its own denormalized copy of the data it needs to read (the Orders service keeps product names and prices it heard about via ProductUpdated events). Reads are fast and local; the copy is eventually consistent.

CQRS

Command Query Responsibility Segregation separates the model used for writes (normalized, enforcing invariants) from one or more models used for reads (denormalized, shaped for each screen, possibly in a different store such as a search index or cache). Writes produce events; projections update the read models. CQRS lets reads and writes scale independently and use the best storage for each, at the cost of eventual consistency and more moving parts. Use it for the parts of a system where read and write needs genuinely diverge, not everywhere.

6.8

Migrating Safely: the Strangler Fig Pattern

"Rewrite the monolith as microservices" big-bang projects have a notoriously poor track record: they take years, the old system keeps changing underneath them, and cut-over day is enormously risky. Martin Fowler's strangler fig pattern (named after a vine that gradually grows around a host tree) migrates incrementally instead.

1. Put a routing layer (proxy or gateway) in front of the legacy system.
2. Build one capability as a new service, ideally one with clear boundaries and high value.
3. Route a small percentage of that capability's traffic to it; compare results (some teams run both and diff the responses — "shadow" or "dark" traffic).
4. Shift traffic fully, delete the old code path, and repeat.

Data is the hardest part: the new service needs its own copy, typically kept in sync from the monolith's database via CDC until ownership fully moves.

6.9

Lab: Idempotency Keys & Cursor Pagination

1. Idempotency-key middleware backed by Redis

// app/idempotency.js — replay stored responses for repeated Idempotency-Key headers
function idempotent(cache, { ttlSeconds = 24 * 3600 } = {}) {
  return async (req, res, next) => {
    const key = req.get('Idempotency-Key');
    if (!key) return next();                                  // optional for this lab
    const redisKey = `idem:${req.method}:${req.path}:${key}`;

    // Claim the key atomically. NX = only if absent, so exactly one request wins.
    // Short TTL on the claim: if this process crashes mid-request, the key frees itself.
    const claimed = await cache.set(redisKey, JSON.stringify({ state: 'in_progress' }), { NX: true, EX: 60 });
    if (!claimed) {
      const saved = JSON.parse(await cache.get(redisKey));
      if (saved.state === 'in_progress') return res.status(409).json({ error: 'request with this key is still in progress' });
      return res.status(saved.status).set('Idempotent-Replayed', 'true').json(saved.body);
    }

    // Capture the response so retries get exactly the same answer
    const originalJson = res.json.bind(res);
    res.json = (body) => {
      if (res.statusCode < 500) {
        cache.set(redisKey, JSON.stringify({ state: 'done', status: res.statusCode, body }), { EX: ttlSeconds }).catch(() => {});
      } else {
        cache.del(redisKey).catch(() => {});                  // server error: let the client retry for real
      }
      return originalJson(body);
    };
    next();
  };
}
module.exports = { idempotent };

// app/server.js
const { idempotent } = require('./idempotency');
app.post('/api/orders', idempotent(cache), async (req, res, next) => { /* …Module 05 handler… */ });

KEY=$(uuidgen)
for i in 1 2 3; do
  curl -s -i -X POST localhost:8080/api/orders -H "Idempotency-Key: $KEY" \
       -H 'Content-Type: application/json' -d '{"productId":42,"email":"a@example.com"}' | grep -E 'HTTP|Idempotent|orderId'
done
# All three return the same orderId; the 2nd and 3rd carry "Idempotent-Replayed: true".
docker compose exec postgres psql -U lab -d lab -c "SELECT count(*) FROM orders WHERE email = 'a@example.com';"

2. Cursor pagination

// GET /api/products?limit=20&cursor=eyJpZCI6MjB9
app.get('/api/products', async (req, res, next) => {
  try {
    const limit = Math.min(Number(req.query.limit) || 20, 100);          // always cap page size
    const after = req.query.cursor
      ? JSON.parse(Buffer.from(req.query.cursor, 'base64url').toString()).id : 0;
    const { rows } = await db.query(
      'SELECT id, name, price_cents FROM products WHERE id > $1 ORDER BY id LIMIT $2', [after, limit + 1]);
    const hasMore = rows.length > limit;
    const items = rows.slice(0, limit);
    const nextCursor = hasMore
      ? Buffer.from(JSON.stringify({ id: items.at(-1).id })).toString('base64url') : null;
    res.json({ items, next_cursor: nextCursor });
  } catch (err) { next(err); }
});

Fetching limit + 1 rows tells you whether another page exists without a separate COUNT(*) — which on a large table is itself an expensive scan. The opaque, encoded cursor lets you change its contents later (e.g. sort by date + id) without breaking clients.

6.10

Key Takeaways

Next up, Module 07 — Frontend Performance at Scale: the half of every page load that happens on the user's device.

Checkpoint

Module 06 Quiz

Covers architecture styles, distribution costs, API design, gateways, meshes, data ownership and migration.

7.1

Why the Frontend Matters at Scale

A backend that answers in 50 ms is wasted if the browser then spends four seconds downloading, parsing and executing JavaScript before the user sees anything. For most websites, the majority of a page load's time is spent after the HTML response arrives. And unlike your servers, you can't scale the user's device: the median visitor may be on a mid-range phone, on a congested mobile network, with a CPU several times slower than your laptop.

Frontend performance is also a scaling concern on your side: every byte you ship is multiplied by millions of page views in CDN bandwidth, and every page that renders on the server costs compute per visit.

Core Web Vitals

Google's Core Web Vitals are the industry-standard user-centric metrics. Each is judged at the 75th percentile of real page loads, separately for mobile and desktop:

Supporting metrics include TTFB (time to first byte — your backend, CDN and network) and FCP (first contentful paint).

7.2

The Critical Rendering Path

To show anything, the browser must: download the HTML, parse it into the DOM, download and parse CSS into the CSSOM, combine them into a render tree, lay it out and paint it. Some resources block that process:

• CSS is render-blocking: the browser won't paint until the stylesheets in <head> are loaded, to avoid a flash of unstyled content.
• Classic <script> is parser-blocking: HTML parsing stops while the script downloads and executes. defer downloads in parallel and runs after parsing (in order); async runs as soon as it arrives (in any order). Module scripts are deferred by default.
• Late discovery: resources referenced from CSS (fonts, background images) or injected by JavaScript are found late, so they start downloading late.

Interactive: optimize a page load

A simplified model of a product page on a mobile connection (≈150 ms round trips). Toggle optimizations and watch the waterfall and LCP change.

Resource hints

<!-- Open the connection to an origin you'll need soon (DNS + TCP + TLS ahead of time) -->
<link rel="preconnect" href="https://cdn.example.com" crossorigin>

<!-- Fetch a critical, late-discovered resource now -->
<link rel="preload" href="/fonts/inter-var.woff2" as="font" type="font/woff2" crossorigin>

<!-- Tell the browser which image is the LCP element -->
<img src="/img/hero-800.avif" fetchpriority="high" width="800" height="450" alt="…">

<!-- Non-critical JS: download in parallel, run after parsing -->
<script src="/assets/app.3f9a2c.js" defer></script>

7.3

Shipping Less JavaScript

Byte for byte, JavaScript is the most expensive resource on the web. An image of 200 KB is downloaded and decoded; 200 KB of JavaScript is downloaded, parsed, compiled and executed on the main thread — the same thread that handles user input. On a low-end phone, executing a large bundle can take seconds, delaying both rendering (LCP) and interactivity (INP).

• Code splitting: ship only what the current route needs; load the rest on demand with dynamic import().
• Tree shaking: bundlers drop unused exports — if you import specific functions rather than whole libraries.
• Audit dependencies: a date library or icon pack can quietly add hundreds of KB. Tools like bundle analyzers show exactly where bytes come from.
• Compress: Brotli typically makes text assets noticeably smaller than gzip; serve pre-compressed static files from the CDN.
• Tame third-party scripts: analytics, ads, chat widgets and tag managers often outweigh your own code. Load them deferred, after interaction, or not at all.
• Set a performance budget (e.g. ≤ 170 KB of compressed JS on the critical path) and fail the CI build when a change exceeds it.

// Load the heavy chart library only when the user opens the analytics tab
tabButton.addEventListener('click', async () => {
  const { renderChart } = await import('./charts.js');   // separate chunk, fetched on demand
  renderChart(document.querySelector('#chart'), data);
});

7.4

Rendering Strategies and Their Scaling Profiles

Where HTML gets generated determines both user-perceived speed and how much server capacity each page view consumes.

A static (SSG) page is the ultimate scalable website: a launch-day traffic spike is just CDN traffic. Many sites mix strategies per route — static marketing pages, ISR product pages, SSR for personalized dashboards — and use islands or partial hydration so only the interactive widgets on a page ship JavaScript.

7.5

Images, Fonts & Layout Stability

Images are usually the largest bytes on a page and very often the LCP element.

• Modern formats: AVIF and WebP are usually substantially smaller than JPEG/PNG at similar quality. Serve them with fallbacks via <picture> or content negotiation.
• Responsive sizes: don't send a 2400-pixel image to a 400-pixel-wide phone. srcset + sizes let the browser pick.
• Lazy-load below the fold with loading="lazy" — but never the LCP image, which should load eagerly with fetchpriority="high".
• Reserve space: always set width and height (or CSS aspect-ratio) so the layout doesn't jump when the image arrives (CLS).
• Image CDNs resize, convert and cache images on the fly from one high-resolution original.

<img
  src="/img/product-800.webp"
  srcset="/img/product-400.webp 400w, /img/product-800.webp 800w, /img/product-1600.webp 1600w"
  sizes="(max-width: 600px) 100vw, 50vw"
  width="800" height="800"
  loading="lazy" decoding="async"
  alt="Blue running shoe, side view">

Web fonts

Custom fonts are discovered late (inside CSS) and can hide text while loading. Self-host and preload the one or two critical font files, subset them to the characters you need, use font-display: swap (or optional) so text is visible immediately, and tune fallback font metrics to minimize the shift when the web font arrives.

7.6

Main-Thread Work & Responsiveness (INP)

The browser's main thread runs your JavaScript, calculates styles, does layout and paints — one thing at a time. A long task (anything over 50 ms) blocks it: if the user taps during a 300 ms task, nothing visibly happens until the task ends. INP measures exactly this delay plus the time to process the event and paint the result.

• Break up long tasks: process work in chunks and yield to the browser between them (await scheduler.yield() where supported, or setTimeout), so input can be handled in between.
• Move heavy computation off the main thread into a Web Worker (parsing big JSON, image processing, search indexing).
• Render less: virtualize long lists (render only the ~20 visible rows of a 10,000-row table), and avoid re-rendering whole trees on every keystroke.
• Avoid layout thrashing: reading layout (offsetHeight) right after writing styles in a loop forces repeated synchronous layouts. Batch reads, then writes.
• Give immediate feedback: update the UI first (button state, spinner) and do the expensive work afterward.

7.7

Network Efficiency, Caching & Service Workers

Everything from Modules 02–03 applies to the frontend: HTTP/2 and HTTP/3 multiplex many requests over one connection, CDNs shorten round trips, and fingerprinted assets with long Cache-Control lifetimes make repeat visits nearly free. A few frontend-specific points:

• Fewer origins: each new domain (fonts host, analytics, image CDN) costs DNS + TCP + TLS. Self-host critical assets on your main origin or preconnect to the rest.
• Avoid request waterfalls: component A renders, then fetches; then component B renders, then fetches. Fetch data in parallel at the route level, or on the server.
• Prefetch likely next pages during idle time (on hover or viewport visibility) so navigation feels instant.

Service workers

A service worker is a script that sits between the page and the network, able to answer requests from its own cache. It enables offline support and instant repeat loads with strategies such as:

7.8

Measuring: Lab Data vs Field Data

Lab data (Lighthouse, WebPageTest, browser DevTools) runs a page load in a controlled environment: reproducible and great for debugging, but it's one device on one network. Field data — Real User Monitoring (RUM) — collects metrics from actual visitors, capturing the true spread of devices, networks and behaviors. Chrome's public CrUX dataset provides field data for popular sites. Optimize with lab tools; judge success with field data at p75.

// Real User Monitoring with Google's web-vitals library
import { onLCP, onINP, onCLS, onTTFB } from 'web-vitals';

function report(metric) {
  const body = JSON.stringify({
    name: metric.name, value: metric.value, rating: metric.rating,
    id: metric.id, page: location.pathname, conn: navigator.connection?.effectiveType,
  });
  // sendBeacon survives page unload; fall back to fetch keepalive
  if (!navigator.sendBeacon?.('/rum', body)) fetch('/rum', { method: 'POST', body, keepalive: true });
}

onLCP(report); onINP(report); onCLS(report); onTTFB(report);

7.9

Lab: Audit and Fix a Page

1. Serve a deliberately slow page from scale-lab

<!-- app/public/slow.html -->
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Product</title>
  <script src="https://cdn.jsdelivr.net/npm/lodash@4.17.21/lodash.min.js"></script>   <!-- parser-blocking -->
  <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter&display=block">
</head>
<body style="font-family: Inter">
  <div id="banner"></div>
  <h1>Trail Runner 3</h1>
  <img src="https://picsum.photos/2400/1600" alt="Shoe">   <!-- huge, no dimensions -->
  <script>
    setTimeout(() => { banner.innerHTML = '<p style="padding:40px;background:#fe0">Free shipping!</p>'; }, 1500);
  </script>
</body>
</html>

// app/server.js
app.use(express.static('public'));

2. Measure

Rebuild, open http://localhost:8080/slow.html in Chrome, then DevTools → Lighthouse → Mobile → Analyze. Or from the terminal:

npx lighthouse http://localhost:8080/slow.html --form-factor=mobile --only-categories=performance --view

Note LCP, CLS and Total Blocking Time, and read the "Diagnostics" section.

3. Fix, one change at a time, re-measuring after each

1. Move the script to the end and add defer (or delete it — nothing uses it).
2. Change display=block to display=swap and add a preconnect to fonts.gstatic.com.
3. Request an appropriately sized image (/800/533), add width/height and fetchpriority="high".
4. Reserve the banner's space (min-height) or render it as an overlay.

Record each run in your RESULTS.md. In DevTools → Performance, enable CPU throttling (4× or 6×) to feel how a mid-range phone experiences the page.

7.10

Key Takeaways

Next up, Module 08 — Containers, Orchestration & Autoscaling: running all these services reliably, and resizing the fleet automatically.

Checkpoint

Module 07 Quiz

Covers Web Vitals, the critical rendering path, JavaScript cost, rendering strategies, media, main-thread work and measurement.

8.1

Why Containers?

Horizontal scaling and autoscaling assume you can start an identical copy of your service anywhere, quickly, any number of times. Before containers, that meant hand-configured servers that slowly drifted apart ("it works on server 3 but not server 7"). A container image packages your code with its runtime, libraries and OS-level dependencies into one immutable artifact. A container is a running instance of it, isolated with Linux namespaces and resource-limited with cgroups (which you already used in the lab's cpus: "1.0").

The twelve-factor app, in brief

The "twelve-factor" methodology (from Heroku engineers, 2011) captures what makes an app easy to run as interchangeable copies. The factors that matter most for scaling:

• Config in the environment, not in the code or image — the same image runs in staging and production.
• Stateless processes; persistent state lives in backing services (databases, caches, object storage) — Module 01.
• Disposability: fast startup and graceful shutdown on SIGTERM — Module 02's draining.
• Logs as event streams written to stdout; the platform collects them — Module 09.
• Scale out via the process model: run more copies rather than bigger ones.
• Dev/prod parity: the same image everywhere; build once, promote through environments.

8.2

Building Good Images

Image size and build design directly affect how fast you can scale out (every new node must pull the image) and how safely you can run.

# ---- build stage: has compilers, dev dependencies, source ----
FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci                          # cached until package*.json changes
COPY . .
RUN npm run build && npm prune --omit=dev

# ---- runtime stage: only what's needed to run ----
FROM node:22-alpine
ENV NODE_ENV=production
WORKDIR /app
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER node                           # never run as root
EXPOSE 3000
CMD ["node", "dist/server.js"]      # exec form: receives SIGTERM directly

• Layer caching: each instruction is a cached layer. Copy dependency manifests and install before copying source, so code changes don't reinstall dependencies.
• Multi-stage builds keep compilers and dev dependencies out of the final image — often shrinking it by 5–10×.
• Small, minimal bases (alpine, slim, distroless) pull faster and contain fewer vulnerable packages.
• Pin versions (and for full reproducibility, image digests), scan images for known vulnerabilities in CI, and add a .dockerignore so node_modules, .git and secrets never enter the build context.

8.3

Kubernetes Core Concepts

Docker Compose runs containers on one machine. An orchestrator runs them across a fleet: it decides which machine each container runs on, restarts failures, connects services, rolls out new versions and scales replicas. Kubernetes (K8s) is the de-facto standard, available as a managed service on every major cloud (EKS, GKE, AKS).

Desired state and reconciliation

You don't tell Kubernetes what to do; you declare what you want ("3 replicas of image v42, each with 0.5 CPU"). Controllers run endless reconciliation loops: observe the actual state, compare it to the desired state, and act to close the gap. A crashed pod isn't "restarted by a script" — the controller simply notices 2 ≠ 3 and creates another.

8.4

Resources: Requests, Limits & Their Surprises

Every container should declare how much CPU and memory it needs. Two numbers, with very different meanings:

• Requests are what the scheduler reserves. A pod is only placed on a node with enough unreserved capacity. Requests also drive autoscaling percentages ("CPU at 70% of request").
• Limits are hard caps enforced at runtime. Exceeding the CPU limit causes throttling — the container is paused for the rest of each scheduling period, which shows up as mysterious latency spikes. Exceeding the memory limit gets the container killed (OOMKilled) and restarted.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scale-lab
spec:
  replicas: 3
  selector:
    matchLabels: { app: scale-lab }
  template:
    metadata:
      labels: { app: scale-lab }
    spec:
      containers:
        - name: app
          image: scale-lab-app:v42
          ports: [{ containerPort: 3000 }]
          resources:
            requests: { cpu: 250m, memory: 192Mi }   # 250 millicores = ¼ core reserved
            limits:   { memory: 256Mi }              # memory cap; no CPU limit (see below)
          envFrom:
            - configMapRef: { name: scale-lab-config }
            - secretRef:    { name: scale-lab-secrets }

Kubernetes assigns a Quality of Service class from these settings: Guaranteed (requests equal limits for every resource), Burstable (some requests set), BestEffort (nothing set). Under memory pressure, BestEffort pods are evicted first. Running production workloads without requests is asking the scheduler to pack nodes blindly.

8.5

Probes, Lifecycle & Spreading Across Failure Domains

Module 02's health checks and graceful draining map directly onto Kubernetes features:

          readinessProbe:            # in the Service's endpoints only while this passes
            httpGet: { path: /ready, port: 3000 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:             # restart the container if this fails repeatedly
            httpGet: { path: /health, port: 3000 }
            periodSeconds: 10
            failureThreshold: 3
          startupProbe:              # give slow starters time before liveness kicks in
            httpGet: { path: /health, port: 3000 }
            periodSeconds: 2
            failureThreshold: 30     # up to 60 s to start
          lifecycle:
            preStop:                 # keep serving briefly while endpoints update
              sleep: { seconds: 5 }  # (older clusters: exec: { command: ["sleep", "5"] })
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:     # don't put all replicas in one zone
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: scale-lab } }

A PodDisruptionBudget (e.g. minAvailable: 2) stops voluntary disruptions — node upgrades, cluster autoscaler scale-downs — from evicting too many replicas at once. Combined with zone spreading, it means a node drain or even a zone outage leaves enough capacity serving.

8.6

Autoscaling: Pods, Nodes and Event-Driven Scaling

Autoscaling in Kubernetes happens at two levels that must work together:

• Horizontal Pod Autoscaler (HPA) changes the replica count of a Deployment based on metrics: CPU or memory utilization relative to requests, or custom metrics such as requests per second or queue depth.
• Node autoscaling (Cluster Autoscaler, or Karpenter on AWS) adds nodes when pods can't be scheduled for lack of capacity, and removes underused nodes.
• Vertical Pod Autoscaler (VPA) recommends or adjusts requests/limits for right-sizing — useful for tuning, usually not combined with HPA on the same metric.
• KEDA (Kubernetes Event-Driven Autoscaling) scales on external signals — Kafka lag, SQS depth, Redis list length — including scaling workers to zero when the queue is empty (Module 05).

The HPA formula

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: scale-lab
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: scale-lab }
  minReplicas: 3                   # never below N+1 across zones
  maxReplicas: 30                  # protect the database (and the budget)
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }   # the hockey stick!
  behavior:
    scaleUp:
      policies: [{ type: Percent, value: 100, periodSeconds: 30 }]   # at most double every 30 s
    scaleDown:
      stabilizationWindowSeconds: 300                                # avoid flapping

Scaling isn't instant

From a traffic spike to new capacity serving requests, the delays stack up: metrics are collected (~15–60 s), the HPA reacts, pods are scheduled, images pulled, the app starts and passes readiness. If no node has room, the node autoscaler must also boot a new machine (often 1–3+ minutes). A sudden spike can therefore hit your existing capacity for several minutes before help arrives.

8.7

Deployment Strategies: Rolling, Blue-Green & Canary

As Module 01 noted, most outages are caused by changes. How you roll out a new version determines how many users a bad release can hurt and how fast you can undo it.

The gold standard is an automated canary analysis: the deployment pauses at each step, compares the canary's error rate and latency with the stable version's, and rolls back automatically if they're worse. Remember that the database is shared by old and new versions during any rollout — schema changes must follow Module 06's expand → migrate → contract sequence so both versions work at every step.

8.8

Serverless & Managed Containers

Functions-as-a-Service (AWS Lambda, Google Cloud Functions, Azure Functions, Cloudflare Workers) take autoscaling to its extreme: you upload code, and the platform runs one instance per concurrent request, scaling from zero to thousands with no servers, nodes or HPAs to manage. You pay per invocation and compute time. Serverless containers (Cloud Run, AWS Fargate, Azure Container Apps) offer similar elasticity for ordinary container images.

8.9

Lab: Autoscale scale-lab on a Local Kubernetes Cluster

We'll use kind (Kubernetes in Docker) to run a real cluster on your laptop.

1. Create the cluster and install metrics-server

brew install kind kubectl           # Linux/WSL: see kind.sigs.k8s.io and kubernetes.io for install steps
kind create cluster --name scale-lab

# metrics-server feeds CPU metrics to the HPA. On kind it needs --kubelet-insecure-tls.
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system patch deployment metrics-server --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

2. Load the image and deploy

docker build -t scale-lab-app:v1 ./app
kind load docker-image scale-lab-app:v1 --name scale-lab     # no registry needed

# The app connects to Redis at startup, so give it one
kubectl create deployment redis --image=redis:7-alpine
kubectl expose deployment redis --port=6379

# k8s/app.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: scale-lab }
spec:
  replicas: 1
  selector: { matchLabels: { app: scale-lab } }
  template:
    metadata: { labels: { app: scale-lab } }
    spec:
      containers:
        - name: app
          image: scale-lab-app:v1
          env:
            - { name: REDIS_URL, value: "redis://redis:6379" }
            - { name: DATABASE_URL, value: "postgres://unused:unused@nowhere:5432/none" }  # /api/cpu needs no DB
          resources: { requests: { cpu: 200m, memory: 128Mi }, limits: { memory: 256Mi } }
          readinessProbe: { httpGet: { path: /health, port: 3000 }, periodSeconds: 5 }
---
apiVersion: v1
kind: Service
metadata: { name: scale-lab }
spec:
  selector: { app: scale-lab }
  ports: [{ port: 80, targetPort: 3000 }]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: scale-lab }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: scale-lab }
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 50 } }

kubectl apply -f k8s/app.yaml
kubectl get pods -w        # wait for 1/1 Running, then Ctrl+C

3. Generate load from inside the cluster and watch it scale

# terminal 1 — a simple load generator pod hitting the CPU-burning endpoint
kubectl run load --image=busybox:1.36 --restart=Never -- /bin/sh -c \
  "while true; do wget -q -O- 'http://scale-lab/api/cpu?ms=50' > /dev/null; done"

# terminal 2 — watch utilization and replicas climb (metrics lag ~30–60 s)
kubectl get hpa scale-lab -w

You should see CPU far above the 50% target, then replicas increase following the HPA formula. Stop the load (kubectl delete pod load) and notice that scale-down waits for the 5-minute stabilization window. Try kubectl delete pod on one app pod while loaded and watch the ReplicaSet replace it within seconds — reconciliation in action. Clean up with kind delete cluster --name scale-lab.

8.10

Key Takeaways

Next up, Module 09 — Observability: you can't scale, debug or trust what you can't see.

Checkpoint

Module 08 Quiz

Covers images, Kubernetes objects, resources, probes, autoscaling, rollouts and serverless.

9.1

Monitoring vs Observability

On one server, debugging means SSH-ing in and reading a log file. Across 200 containers, 15 services, 3 databases and a queue — some of which no longer exist because the autoscaler removed them — that's impossible. You need the system to report on itself.

Monitoring answers questions you knew to ask in advance: "Is error rate above 1%?" Observability is the ability to answer questions you didn't anticipate — "Why are checkout requests from Android users in Brazil slow since 14:05, but only for carts with more than 20 items?" — from the telemetry the system already emits, without shipping new code.

The three pillars (and friends)

Continuous profiles (which functions use CPU and memory) and RUM (Module 07) complete the picture. The real power comes from correlating them: jumping from a latency spike on a dashboard to example traces, and from a slow span to its logs.

9.2

Structured Logging

A log line like User 42 failed to pay order 981 after 3s is easy for humans and painful for machines: to find all payment failures over 2 seconds, you need fragile regular expressions. Structured logs are emitted as JSON with consistent fields, so log platforms can index, filter and aggregate them like a database.

// npm install pino pino-http
const pino = require('pino');
const pinoHttp = require('pino-http');
const { randomUUID } = require('crypto');

const logger = pino({ level: process.env.LOG_LEVEL || 'info', base: { service: 'scale-lab', instance: INSTANCE } });

app.use(pinoHttp({
  logger,
  genReqId: (req, res) => {
    const id = req.get('X-Request-Id') || randomUUID();   // reuse the edge's ID if present
    res.set('X-Request-Id', id);
    return id;
  },
}));

app.post('/api/payments', async (req, res) => {
  // req.log is a child logger: every line carries this request's ID automatically
  req.log.warn({ orderId: 981, provider: 'acme-pay', durationMs: 3021, attempt: 3 }, 'payment failed');
  // …
});

{"level":40,"time":1792234567890,"service":"scale-lab","instance":"a1b2c3","req":{"id":"7f3c9a2e-…","method":"POST","url":"/api/payments"},"orderId":981,"provider":"acme-pay","durationMs":3021,"attempt":3,"msg":"payment failed"}

• Correlation IDs: generate a request ID at the edge, pass it along in a header to every downstream call and queue message, and include it in every log line. One ID then pulls up the whole story across services.
• Log levels: error (someone should look), warn (unexpected but handled), info (significant business events), debug (off in production, switchable at runtime).
• Write to stdout and let the platform ship logs (Fluent Bit, Vector, the cloud's agent) to a central store (Elasticsearch/OpenSearch, Loki, a vendor).

9.3

Metrics & Prometheus

Metrics are the backbone of dashboards and alerts: compact numeric time series, cheap enough to keep for months. Three core types:

• Counter — only goes up (total requests, total errors). You almost always look at its rate per second.
• Gauge — goes up and down (queue depth, memory in use, active connections).
• Histogram — counts observations in buckets (request durations). Lets you compute percentiles across all instances — recall from Module 01 that percentiles can't be averaged, but histograms can be summed.

Prometheus, the de-facto open-source standard, pulls ("scrapes") a /metrics endpoint from every target every 15–30 seconds and stores the series; PromQL queries them; Grafana charts them. Each series is identified by a name plus labels: http_requests_total{route="/api/orders", status="500"}.

// npm install prom-client
const client = require('prom-client');
client.collectDefaultMetrics();      // CPU, memory, event-loop lag, GC…

const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request latency',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on('finish', () => end({ method: req.method, route: req.route?.path || 'unmatched', status: res.statusCode }));
  next();
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

PromQL you'll use daily

# Request rate (RED: Rate), per route, over the last 5 minutes
sum by (route) (rate(http_request_duration_seconds_count[5m]))

# Error ratio (RED: Errors)
sum(rate(http_request_duration_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))

# p99 latency across ALL instances (RED: Duration) — computed from summed histogram buckets
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

What to graph: the golden signals

Google's SRE book recommends four signals for every user-facing service: latency (of successful and failed requests, separately), traffic, errors and saturation. They map neatly onto Module 01's RED (services) and USE (resources) methods. One dashboard row per service, with these four panels, answers "is it us, and where?" in seconds.

9.4

Distributed Tracing & OpenTelemetry

Metrics tell you checkout p99 jumped to 2 seconds. They can't tell you which of the eleven calls inside checkout got slow. A trace records one request's journey as a tree of spans — each span a timed operation (an HTTP handler, a DB query, an outbound call) with attributes and a parent.

Context propagation

For spans from different services to join one trace, each outgoing call carries the trace context. The W3C Trace Context standard defines the traceparent header: 00-<trace-id>-<parent-span-id>-<flags>. Instrumentation libraries read it on incoming requests and add it to outgoing ones — including into queue messages, so asynchronous work joins the same trace.

OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral CNCF standard for generating and exporting traces, metrics and logs. Its auto-instrumentation packages hook into popular libraries (HTTP servers and clients, Express, pg, Redis, gRPC) so you get useful traces without touching every function. Data is exported via the OTLP protocol to a collector and on to any backend: Jaeger, Grafana Tempo, Zipkin, or commercial tools.

Sampling

Tracing every request at high volume is expensive. Head sampling decides at the start (keep 1% of traces): cheap, but it throws away 99% of the rare slow or failed requests you most want. Tail sampling decides after the trace completes, in a collector — keep all errors, all slow traces, and a small share of the rest — at the cost of buffering spans.

9.5

Alerting on SLOs: Error Budgets & Burn Rates

Bad alerting is a scaling problem for people. Alerting on every cause ("CPU > 80%", "one pod restarted", "disk 70% full") at fleet scale produces hundreds of pages a week, most of which don't affect users. On-call engineers learn to ignore them — and then miss the real one. The fix: alert on symptoms users feel, measured against your SLOs (Module 01), and send everything else to dashboards or tickets.

Burn rate

With a 99.9% SLO over 30 days, your error budget is 0.1% of requests. The burn rate is how fast you're spending it: a burn rate of 1 uses exactly the whole budget in 30 days; a burn rate of 14.4 would exhaust it in about 2 days (30 ÷ 14.4 ≈ 2.1). Burn-rate alerts page when the budget is disappearing fast, regardless of the absolute error rate.

The Google SRE Workbook recommends multi-window, multi-burn-rate alerts, for example:

The long window makes the alert significant; the short window makes it reset quickly once the problem is fixed. Every page should link to a runbook: what the alert means, how to confirm it, and first mitigation steps.

9.6

A Debugging Workflow, Profiling & Capacity Reviews

1. Start from the symptom: which SLO is burning, since when, how badly?
2. Scope it: break the RED metrics down by route, region, version, customer tier. Did it start with a deploy or config change? (Annotate deploys on dashboards.)
3. Find the slow or failing component: open exemplar traces for slow requests and look for the long or erroring span.
4. Explain it: read the logs for that span's request ID; check USE metrics of the implicated resource (pool saturation? CPU throttling? replication lag?).
5. Mitigate first, fix later: roll back, shed load, scale up, fail over — then find the root cause calmly.

Continuous profiling

When a service is CPU-bound, profiles show which code is responsible. Continuous profilers (Grafana Pyroscope, Parca, cloud vendors' profilers) sample stacks in production with low overhead and render flame graphs; comparing before/after a deploy often pinpoints a regression to a single function.

Capacity planning as a habit

Use your metrics to look ahead, not just back: track utilization trends against limits (CPU, connections, disk, queue throughput, cloud quotas), load-test before big events using production-like data, and review headroom regularly. The back-of-envelope method from Module 01, fed with real numbers, is your forecasting model.

9.7

Incident Response & Blameless Postmortems

Observability pays off most in the worst hours. Organizations that handle incidents well share a few practices:

• Clear roles: an incident commander coordinates and decides; others investigate, communicate with stakeholders, and scribe the timeline. The commander doesn't debug.
• Severity levels agreed in advance, with matching response expectations.
• Communicate early: a status page update within minutes beats a perfect explanation after an hour.
• Measure response: time to detect (MTTD) and time to recover (MTTR) often matter more than how rarely things fail.
• Blameless postmortems: a written analysis of what happened, why the system allowed it, and concrete follow-up actions — focused on systems and processes, not individuals, so people share what really happened.

9.8

Lab: Metrics with Prometheus & Grafana, Traces with Jaeger

1. Instrument the app

Add the prom-client code from 9.3 to app/server.js. Then add tracing in a separate file loaded before everything else:

cd app && npm install prom-client @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http && cd ..

// app/tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

new NodeSDK({
  serviceName: 'scale-lab',
  traceExporter: new OTLPTraceExporter({ url: 'http://jaeger:4318/v1/traces' }),
  instrumentations: [getNodeAutoInstrumentations()],   // http, express, pg, redis, …
}).start();

# app/Dockerfile — change the start command
CMD ["node", "--require", "./tracing.js", "server.js"]

2. Add the observability stack to Compose

  prometheus:
    image: prom/prometheus:v2.53.0
    volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml:ro"]
    ports: ["9090:9090"]

  grafana:
    image: grafana/grafana:11.1.0
    ports: ["3001:3000"]          # login admin / admin on first run

  jaeger:
    image: jaegertracing/all-in-one:1.58
    environment: { COLLECTOR_OTLP_ENABLED: "true" }
    ports: ["16686:16686"]        # Jaeger UI

# prometheus.yml — discover every replica of "app" through Docker's DNS
scrape_configs:
  - job_name: scale-lab
    scrape_interval: 5s
    dns_sd_configs:
      - names: ["app"]
        type: A
        port: 3000

3. Generate traffic and explore

docker compose up -d --build --scale app=3
k6 run -e CACHED=1 k6/cache.js

• Prometheus (localhost:9090) → Status → Targets: three healthy app targets. Run the p99 query from 9.3.
• Grafana (localhost:3001): add a Prometheus data source at http://prometheus:9090, then build a dashboard with rate, error ratio and p99 panels.
• Jaeger (localhost:16686): pick service scale-lab, find traces, and compare a cache hit (an HTTP span plus a Redis span) with a miss (HTTP + Redis + pg spans).

Finally, stop Redis (docker compose stop redis) during a run and watch the incident unfold on all three tools: the error ratio or latency panel moves first, traces show missing or failing Redis spans, and logs explain why.

9.9

Key Takeaways

Next up, Module 10 — Resilience Engineering: designing the system so that when (not if) things fail, users barely notice.

Checkpoint

Module 09 Quiz

Covers logs, metrics, traces, SLO alerting, debugging and incident response.

10.1

Failure Is Normal: How Outages Cascade

At scale, something is always broken: a disk, a node, a zone's network, a dependency's p99. Resilience engineering accepts this and asks a different question: when a part fails, how do we stop that failure from spreading? Most large outages aren't one component dying; they're a small failure cascading through a system that amplified it.

Anatomy of a cascade

1. A dependency (say, the recommendations service) slows from 50 ms to 5 s.
2. Callers wait on it. Their worker threads, connections and memory fill with stuck requests (Little's Law: in-flight = rate × latency, and latency just went up 100×).
3. The callers now can't serve anything — including requests that never needed recommendations. Their health checks fail.
4. Clients time out and retry, multiplying load on everything. Autoscaling adds instances that open more connections to the struggling database.
5. One slow, non-critical feature has taken down checkout.

The defenses form layers: timeouts bound waiting, retries recover from blips (carefully), circuit breakers stop calling what's broken, bulkheads contain damage, rate limiting and load shedding protect capacity, and redundancy survives losing whole locations.

10.2

Timeouts, Deadlines & Disciplined Retries

Every network call needs a timeout. Many client libraries default to none (or minutes), which means one hung dependency can hold your resources forever. Base the timeout on the dependency's observed latency — e.g. a bit above its p99.9 — not on a round number.

Deadline propagation

Better than independent timeouts at each hop is a deadline: the edge decides "this request must finish within 2 s", and passes the remaining time to each downstream call (gRPC does this natively). A service receiving a request with 30 ms left doesn't start a 500 ms query; it fails fast, and nobody wastes work on an answer the user will never see.

// Node 18+: abort an outbound call when the request's deadline passes
async function callWithDeadline(url, deadlineMs, init = {}) {
  const remaining = deadlineMs - Date.now();
  if (remaining <= 0) throw new Error('deadline exceeded before call');
  return fetch(url, { ...init, signal: AbortSignal.timeout(remaining) });
}

Retries, revisited

• Retry only idempotent operations (or use idempotency keys — Module 06) and only on retryable errors (timeouts, 503, connection resets — not 400 or 404).
• Use exponential backoff with jitter (Module 05).
• Retry at one layer of the stack, not every layer (Module 02's 27×).
• Enforce a retry budget: retries may add at most ~10% extra load. When most requests are failing, retries can't help and only amplify the outage.

10.3

Circuit Breakers

A timeout limits how long one call waits. But if a dependency is clearly down, making every request wait for the timeout is still wasteful — and keeps hammering a service that's trying to recover. A circuit breaker tracks recent failures and, past a threshold, stops calling the dependency entirely for a while, failing instantly (ideally with a fallback).

// npm install opossum
const CircuitBreaker = require('opossum');

async function fetchRecommendations(userId) {
  const res = await fetch(`http://recs/api/users/${userId}/recs`, { signal: AbortSignal.timeout(300) });
  if (!res.ok) throw new Error(`recs ${res.status}`);
  return res.json();
}

const recsBreaker = new CircuitBreaker(fetchRecommendations, {
  timeout: 300,                   // treat slow calls as failures
  errorThresholdPercentage: 50,   // open when ≥50% of recent calls fail…
  volumeThreshold: 20,            // …but only after at least 20 calls in the window
  resetTimeout: 30_000,           // stay open 30 s, then go half-open
});
recsBreaker.fallback(() => ({ items: [], degraded: true }));   // the page still renders
recsBreaker.on('open', () => logger.warn('recs circuit OPEN'));

app.get('/api/home', async (req, res) => {
  const recs = await recsBreaker.fire(req.user.id);   // never throws while a fallback exists
  res.json({ hero: await getHero(), recs });
});

10.4

Bulkheads & Isolation

Ships are divided into watertight compartments (bulkheads) so a hull breach floods one section, not the whole ship. In software, a bulkhead gives each dependency or workload its own bounded pool of resources, so exhausting one can't starve the others.

Bulkheads exist at every scale:

• In code: separate connection pools or concurrency limits per dependency (a semaphore of 10 for the recommendations client).
• In deployment: separate instance pools for critical vs non-critical traffic, or for internal batch jobs vs user traffic; separate queues per tenant tier.
• In infrastructure: cell-based architecture — the whole stack replicated into independent cells, each serving a subset of customers. A bad deploy or a poison request affects one cell's customers, not everyone. Shuffle sharding (assigning each customer to a random combination of cells or nodes) shrinks the overlap between customers even further.

10.5

Rate Limiting

Rate limiting caps how many requests a client (user, API key, IP, tenant) may make per unit of time. It protects capacity from abusive or buggy clients, enforces fair use between tenants, and backs commercial API plans. The algorithms differ in how they treat bursts:

Interactive: token bucket

Distributed rate limiting with Redis

With many app instances, per-instance counters don't add up to a global limit. A shared Redis with an atomic Lua script does:

// Token bucket in Redis: atomic because Redis runs a Lua script without interleaving
const TOKEN_BUCKET = `
local key, capacity, rate, now = KEYS[1], tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local b = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(b[1]) or capacity
local ts = tonumber(b[2]) or now
tokens = math.min(capacity, tokens + (now - ts) / 1000 * rate)   -- refill since last call
local allowed = 0
if tokens >= 1 then tokens = tokens - 1; allowed = 1 end
redis.call('HSET', key, 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', key, math.ceil(capacity / rate * 1000) + 1000)
return { allowed, math.floor(tokens) }`;

function rateLimit({ capacity = 20, ratePerSec = 5, keyFn = (req) => req.ip } = {}) {
  return async (req, res, next) => {
    try {
      const [allowed, remaining] = await cache.eval(TOKEN_BUCKET, {
        keys: [`rl:${keyFn(req)}`], arguments: [String(capacity), String(ratePerSec), String(Date.now())],
      });
      res.set('X-RateLimit-Remaining', String(remaining));
      if (!allowed) return res.status(429).set('Retry-After', '1').json({ error: 'rate limit exceeded' });
      next();
    } catch {
      next();   // fail open: if Redis is down, don't take the whole API down with it
    }
  };
}

app.use('/api/', rateLimit({ capacity: 20, ratePerSec: 5 }));

10.6

Load Shedding & Graceful Degradation

Rate limiting protects against individual clients. Load shedding protects the server itself when total demand exceeds capacity — say, a flash sale arrives faster than autoscaling can react. The insight from Module 01's queueing theory: an overloaded server that accepts everything serves nobody well, because every request waits in an ever-growing queue and times out. Rejecting some requests quickly lets the rest succeed.

• Bound concurrency and queues: cap in-flight requests per instance (sized with Little's Law); reject with 503 instantly when full.
• Drop work that's already doomed: if a request has waited in a queue longer than its client's timeout, discard it rather than processing it.
• Prioritize: shed analytics beacons and prefetches before page loads, and page loads before checkout. Critical traffic gets reserved capacity.
• Adaptive limits: algorithms inspired by TCP congestion control (e.g. Netflix's concurrency-limits library) raise the concurrency limit while latency stays low and cut it when latency climbs, finding capacity automatically.

// A minimal concurrency limiter for an Express app
const MAX_IN_FLIGHT = 64;           // ≈ instance throughput × target latency (Little's Law)
let inFlight = 0;
app.use((req, res, next) => {
  if (inFlight >= MAX_IN_FLIGHT && !req.path.startsWith('/api/checkout')) {
    return res.status(503).set('Retry-After', '2').json({ error: 'overloaded, please retry' });
  }
  inFlight++;
  res.on('close', () => { inFlight--; });
  next();
});

Graceful degradation

Design features with "off" positions. Under stress, a system can serve stale cached pages, disable expensive features (search suggestions, personalization, live inventory counts), switch to a lightweight static version of the homepage, or put new visitors in a waiting room. Users notice reduced features far less than errors. Some teams run deliberate brownouts — briefly disabling a non-critical dependency in production — to prove the degraded paths really work.

10.7

Redundancy: Multi-AZ, Multi-Region & Disaster Recovery

Cloud regions are divided into availability zones — separate data centers with independent power and networking, close enough for low-latency synchronous replication. Running every tier across at least two (better three) zones is the baseline for production; a single zone failure should be a non-event. Multi-region protects against rarer, larger failures and puts the system closer to global users, at a much higher cost in complexity — mostly because of data.

RPO and RTO

• RPO (Recovery Point Objective): how much data you can afford to lose, measured in time. Asynchronous cross-region replication with 5 s of lag means an RPO of ~5 s.
• RTO (Recovery Time Objective): how long you can afford to be down while recovering.

10.8

Chaos Engineering & Game Days

You can't know whether timeouts, breakers, fallbacks and failover work until they're exercised. Chaos engineering, pioneered at Netflix with Chaos Monkey (which randomly terminated production instances), is the discipline of running controlled experiments that inject failure, to build confidence in the system's behavior.

1. Define steady state in measurable terms (SLIs: success rate, p99).
2. Hypothesize: "If one Redis replica dies, steady state continues."
3. Inject a realistic fault: kill instances, add latency, drop packets, fill disks, block a dependency, fail a zone.
4. Limit the blast radius: start in staging, then a small percentage of production traffic, with an abort switch.
5. Learn and fix: every disproved hypothesis is a weakness found on your schedule rather than at 3 a.m.

Tools range from simple (stopping containers, Toxiproxy for network faults) to platforms (Gremlin, LitmusChaos, AWS Fault Injection Service). A game day is a scheduled team exercise: inject a failure, have on-call respond as if it were real, and evaluate detection, runbooks and communication as well as the software.

10.9

Lab: Inject Database Latency, Then Contain It

1. Put Toxiproxy between the app and PostgreSQL

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.9.0
    ports: ["8474:8474"]                  # control API

  app:
    environment:
      DATABASE_URL: postgres://lab:lab@toxiproxy:15432/lab    # app now talks to the proxy

docker compose up -d --build --scale app=3
# create a proxy: toxiproxy listens on 15432 and forwards to postgres:5432
curl -s -X POST localhost:8474/proxies \
  -d '{"name":"pg","listen":"0.0.0.0:15432","upstream":"postgres:5432"}'

2. Baseline, then inject 2 s of latency

k6 run -e CACHED=0 k6/cache.js                     # baseline
curl -s -X POST localhost:8474/proxies/pg/toxics \
  -d '{"name":"slow","type":"latency","attributes":{"latency":2000}}'
k6 run -e CACHED=0 k6/cache.js                     # watch it fall apart

Note what breaks: latency explodes, pools exhaust (Module 04), and — if /health or other endpoints share the event loop and pool — even unrelated requests slow down. That's a cascade in miniature.

3. Contain it

• Set statement_timeout and connectionTimeoutMillis on the pool (Module 04) so calls fail in ~500 ms instead of hanging.
• Wrap the database read in an opossum breaker with a fallback that serves the cached value (even if stale) or a clear 503.
• Add the concurrency limiter from 10.6 so the instance sheds excess load instead of queueing it.

Re-run with the toxic still active. Success: errors are fast and bounded, cached products keep serving, and when you remove the toxic (curl -X DELETE localhost:8474/proxies/pg/toxics/slow) the breaker half-opens and the system recovers on its own within ~30 s.

10.10

Key Takeaways

Next up, Module 11 — Security at Scale: protecting a large, public, distributed system from people who are trying to break it on purpose.

Checkpoint

Module 10 Quiz

Covers cascades, timeouts, retries, circuit breakers, bulkheads, rate limiting, load shedding, disaster recovery and chaos engineering.

11.1

Why Security Is a Scaling Problem

Everything in this course has assumed load comes from users who want your service to work. At scale, a meaningful share of your traffic comes from people and bots who want it to fail, or to leak data: credential-stuffing bots, scrapers, fraudsters, attackers probing for injection, and botnets launching denial-of-service floods. Growth also expands the attack surface: more services, more APIs, more engineers with access, more dependencies, more secrets.

Threat modeling, briefly

Before choosing controls, ask "what could go wrong?" for each part of the system. Microsoft's STRIDE mnemonic is a handy checklist: Spoofing (pretending to be someone), Tampering (modifying data), Repudiation (denying actions), Information disclosure (leaks), Denial of service, and Elevation of privilege. Run it on every new service or major feature, especially at trust boundaries — where data crosses from the internet into your network, or from one service to another.

11.2

Denial of Service: Attacks & Layered Defenses

A distributed denial-of-service (DDoS) attack tries to exhaust some resource — bandwidth, connection tables, CPU, or an expensive endpoint — using traffic from many sources at once.

11.3

Web Application Firewalls, Bots & Abuse

A WAF inspects HTTP requests at the edge and blocks known attack patterns: SQL injection and cross-site scripting payloads, path traversal, known exploit signatures, oversized bodies, unusual methods. Managed rule sets (e.g. the OWASP Core Rule Set, cloud providers' managed rules) cover the common cases; custom rules encode your application's specifics. A WAF is a safety net — it does not replace writing code that's immune to injection (parameterized queries, output encoding).

The OWASP Top 10 is the standard list of critical web application risks; in its 2021 edition, Broken Access Control ranks first — a reminder that the most common serious flaw is letting users do or see things they shouldn't, which no firewall can detect for you.

Bots

A large share of internet traffic is automated. Some is welcome (search crawlers, monitoring); much isn't:

• Credential stuffing: replaying username/password pairs leaked from other sites against your login, at scale.
• Scraping of prices, content and inventory; scalping of limited products and tickets.
• Fake accounts, spam, and fraud.

Defenses combine signals: rate limits per account and per IP; device and browser fingerprinting; behavioral analysis; IP reputation; challenges (proof-of-work or CAPTCHA-style) for suspicious traffic only; breached-password checks; and multi-factor authentication, which makes stolen passwords far less useful.

11.4

Identity: Authentication & Authorization Across Services

Authentication (AuthN) establishes who is making a request; authorization (AuthZ) decides what they may do. In a monolith, both happen in one place. Across many services, you need a consistent, scalable way to carry identity and enforce policy.

Tokens and OpenID Connect

Most modern systems delegate login to an identity provider using OAuth 2.0 and OpenID Connect (OIDC), which issue signed tokens — commonly JWTs. Any service can verify a JWT's signature locally using the provider's public keys (published at a JWKS endpoint), with no call to a central session store. That scales beautifully, with caveats:

• Validate properly: signature with an allow-listed algorithm, issuer, audience, expiry. Never accept unsigned tokens or trust claims you haven't verified.
• Keep access tokens short-lived (minutes) because they're hard to revoke (Module 01); use refresh tokens for long sessions, and revocation lists for emergencies.
• Rotate signing keys: publish new keys in the JWKS before using them, and cache JWKS with a refresh on unknown key IDs.

Authorization at scale

• RBAC (role-based): users get roles, roles get permissions. Simple; coarse.
• ABAC (attribute-based): rules over attributes of the user, resource and context ("managers in the same region during business hours").
• ReBAC (relationship-based): permissions derived from relationships ("can edit if editor of the parent folder"), as in Google's Zanzibar system and its open-source descendants (OpenFGA, SpiceDB).

Many organizations centralize policy definitions (e.g. Open Policy Agent, or a dedicated authorization service) while enforcing decisions locally in each service. Whatever the model, enforce authorization on the server for every object access — checking that order 981 belongs to the caller, not just that the caller is logged in. Missing object-level checks (often called IDOR or BOLA) top API security risk lists.

11.5

Secrets Management & Zero-Trust Networking

Secrets

Database passwords, API keys and signing keys multiply with every service. Rules that scale:

• Never in source code, images or logs. Scan repositories and CI for leaked secrets (pre-commit hooks, secret-scanning tools) — git history is forever.
• Central secret store (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault), injected at runtime with per-service access policies.
• Short-lived, dynamic credentials where possible: a secret store can mint a database credential valid for one hour for one service, so a leak expires on its own.
• Rotate routinely and automatically; a rotation you've never rehearsed will fail during an incident.
• Envelope encryption with a KMS: data is encrypted with data keys, which are themselves encrypted by a master key that never leaves the key-management service.

Zero trust

The old model — a hard perimeter with a trusted interior — fails once an attacker is inside. Zero trust assumes the network is hostile: every request is authenticated and authorized, regardless of where it comes from.

• Workload identity: each service gets a cryptographic identity (e.g. a SPIFFE ID in an X.509 certificate) instead of being trusted for its IP address.
• Mutual TLS between services, usually automated by a service mesh (Module 06) that issues and rotates short-lived certificates.
• Least privilege: network policies so services can reach only what they need; IAM roles scoped to exactly the resources each service uses.

11.6

Protecting Data & Isolating Tenants

• Encrypt in transit (TLS everywhere, including internal hops) and at rest (database, backups, object storage, queues) — managed services make the latter a checkbox.
• Minimize: the safest personal data is data you never collected. Set retention limits and delete on schedule.
• Tokenize highly sensitive fields (card numbers, national IDs): store them in a dedicated, tightly controlled vault and pass opaque tokens through the rest of the system.
• Keep sensitive data out of side channels: logs, traces, analytics events, error trackers and caches all copy data. Redact at the source.
• Respect data residency requirements (e.g. EU data stays in the EU) — often the reason for geographic sharding (Module 04).

Multi-tenant isolation

SaaS systems serve many customers from shared infrastructure; one missing WHERE tenant_id = ? leaks a customer's data to another. Options range from separate databases per tenant (strongest isolation, highest cost) to shared tables with a tenant column. With shared tables, the database itself can enforce isolation through row-level security, so an application bug can't cross tenants:

ALTER TABLE orders ADD COLUMN tenant_id BIGINT NOT NULL DEFAULT 1;
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON orders
  USING (tenant_id = current_setting('app.tenant_id')::bigint);

-- The app sets the tenant for each transaction, then queries normally:
BEGIN;
SET LOCAL app.tenant_id = '42';
SELECT * FROM orders;          -- only tenant 42's rows, even without a WHERE clause
COMMIT;

11.7

Software Supply Chain & Dependencies

A typical application contains far more third-party code than first-party code, plus base images, build tools and CI actions. Each is a path into production. High-profile incidents — compromised packages, malicious updates, typosquatted names, backdoored build systems — have shown how far one poisoned dependency can spread.

• Lock dependencies (package-lock.json, npm ci) so builds are reproducible and don't silently pick up new versions.
• Scan continuously (software composition analysis) for known vulnerabilities in dependencies and images, and patch on a defined cadence — faster for critical issues.
• Review new dependencies: maintenance activity, popularity, install scripts, transitive footprint. Fewer dependencies is fewer risks.
• Produce an SBOM (software bill of materials) per build, so when the next widespread vulnerability is announced you can answer "are we affected, and where?" in minutes.
• Sign and verify artifacts (e.g. Sigstore/cosign) and only deploy images built by your CI; frameworks like SLSA describe progressively stronger build integrity.
• Harden CI: pin third-party actions to commit hashes, scope CI credentials minimally, and protect the main branch.

11.8

Lab: Security Headers, Login Rate Limits & Row-Level Security

1. Security headers with Helmet

// npm install helmet
const helmet = require('helmet');
app.use(helmet());          // HSTS, X-Content-Type-Options, frame protections, a default CSP, …
app.disable('x-powered-by');
app.set('trust proxy', 1);  // we're behind exactly one proxy (Nginx) — Module 02

curl -sI localhost:8080/health

Compare the response headers before and after. Then scan a public site you own with a headers-checking tool or the browser's DevTools security panel.

2. Rate-limit logins per account and per IP

Reuse the token-bucket limiter from Module 10 with two keys:

app.post('/api/login', express.json(),
  rateLimit({ capacity: 10, ratePerSec: 0.2, keyFn: (req) => `login-ip:${req.ip}` }),
  rateLimit({ capacity: 5,  ratePerSec: 1 / 60, keyFn: (req) => `login-user:${String(req.body?.email).toLowerCase()}` }),
  async (req, res) => {
    // …verify the password with a slow hash (bcrypt/scrypt/argon2), never plain comparison…
    res.status(401).json({ error: 'invalid credentials' });   // same message for unknown user and wrong password
  });

for i in $(seq 1 8); do
  curl -s -o /dev/null -w '%{http_code} ' -X POST localhost:8080/api/login \
    -H 'Content-Type: application/json' -d '{"email":"alice@example.com","password":"guess"}'
done; echo     # 401 ×5, then 429s for that account

3. Row-level security with an unprivileged role

docker compose exec postgres psql -U lab -d lab

-- Run the ALTER TABLE / CREATE POLICY statements from 11.6 first, then:
INSERT INTO orders (product_id, email, tenant_id) VALUES (1, 'a@t1.com', 1), (2, 'b@t2.com', 2);

CREATE ROLE app_user LOGIN PASSWORD 'app_user';          -- not a superuser, not the owner
GRANT SELECT, INSERT, UPDATE ON orders TO app_user;

SET ROLE app_user;
SELECT count(*) FROM orders;                             -- error: app.tenant_id is not set
BEGIN; SET LOCAL app.tenant_id = '2'; SELECT email FROM orders; COMMIT;   -- only b@t2.com
RESET ROLE;                                              -- back to the superuser: sees everything

The error in the first query is a feature: a code path that forgets to set the tenant fails loudly instead of returning everyone's data.

11.9

Key Takeaways

Next up, the finale — Module 12 — Capstone: designing a complete production system for one of the hardest traffic patterns on the web.

Checkpoint

Module 11 Quiz

Covers threat modeling, DDoS, WAFs and bots, identity, secrets, zero trust, data protection and supply chain security.

12.1

A Method for System Design

You've learned the building blocks. Real design work is choosing and combining them under constraints. Whether in a design review or an interview, a consistent method keeps you from jumping straight to boxes and arrows:

1. 1Requirements. Functional (what it does) and non-functional (how much, how fast, how reliable, how consistent). Ask "scale what, to how much, by when?" (Module 01).
2. 2Estimates. Back-of-the-envelope traffic, storage and concurrency. Find the hardest number.
3. 3API & data model. The contract and the entities; where consistency is mandatory.
4. 4High-level architecture. The simplest design that meets the requirements.
5. 5Deep dives. The two or three hardest problems, solved properly.
6. 6Failure modes & operations. What breaks, how you'll know, what happens to users.
7. 7Evolution & cost. What to build first, and what to defer until the numbers demand it.

Our case study is one of the most punishing traffic patterns on the web: a high-demand ticket on-sale. It combines a flash crowd (Modules 02, 03, 08, 10), strict inventory correctness (Module 04), payments and confirmations (Modules 05, 06), bots (Module 11) and the need to see everything in real time (Module 09).

12.2

Requirements & Back-of-the-Envelope Estimates

Functional requirements

• Browse the event page and seat map with live availability.
• Select seats and hold them for 5 minutes while paying.
• Pay; receive confirmation and tickets by email; view tickets in the account.

Non-functional requirements

• Correctness first: never sell the same seat twice, and never charge without issuing a ticket.
• Fairness: people who arrived earlier generally get in earlier; bots don't win.
• Availability: 99.95% for the purchase path during on-sales; the site must not fall over under the crowd.
• Latency: seat hold p99 < 500 ms; page loads LCP < 2.5 s at p75.

Estimates

1. Arrivals: 2 M users, with ~1 M in the first minute → ~17,000 new users/s at peak. Each loads the event page (HTML + assets) and polls status: easily 100,000+ HTTP requests/s at the edge.
2. Buyers: 50,000 seats ÷ ~2.5 tickets per order ≈ 20,000 orders. Only ~1% of visitors can possibly succeed.
3. Inventory writes: holds, releases and purchases for 50,000 seats — perhaps 100,000–200,000 writes over 10 minutes, concentrated on one event.
4. The hardest number: not total traffic (a CDN absorbs that), but the rate of contended writes on a single event's inventory, plus the 17,000 users/s who all want to reach checkout at once.

Use the estimator to see how a waiting room turns that crowd into a manageable flow. It's Little's Law again: if checkout can hold L concurrent shoppers who each take W minutes, you can admit λ = L ÷ W shoppers per minute.

12.3

API & Data Model

GET  /events/{eventId}                      → event details (static, CDN-cached)
GET  /events/{eventId}/availability         → section-level counts (cached ~1–2 s)
GET  /events/{eventId}/sections/{s}/seats   → seat map for one section (cached ~1 s)
POST /events/{eventId}/holds                → { seatIds } → hold for 5 min   [admission token + Idempotency-Key]
POST /orders                                → { holdId, paymentToken } → 202 + orderId   [Idempotency-Key]
GET  /orders/{orderId}                      → status: pending | confirmed | failed
GET  /queue/{eventId}/status                → position, estimated wait (waiting room)

The data splits naturally by how it's accessed and how consistent it must be:

12.4

High-Level Architecture

The guiding idea: make the crowd cheap, and make the scarce resource safe.

• Cheap crowd: event pages are static files on the CDN; seat maps are short-TTL cached JSON; the waiting room is a lightweight page that polls a tiny endpoint. Two million people waiting cost almost nothing.
• Safe scarcity: only admitted users reach the Inventory API; seat state lives in a strongly consistent database with atomic conditional updates; payment and ticketing are asynchronous, idempotent, and recoverable.

12.5

Deep Dives: the Waiting Room and No-Oversell Inventory

Deep dive 1 — the virtual waiting room

Without a waiting room, 17,000 users/s hit the Inventory API at once: the database drowns, everyone gets errors, users retry harder, and the fastest bots win. A waiting room is admission control — Module 10's load shedding, made fair and friendly:

1. Users who arrive before 10:00 wait on a static CDN page. At 10:00, each receives a random position (randomizing early arrivals is fairer than rewarding whoever reloaded fastest) — or first-come order for later arrivals.
2. An admission controller lets users through at a rate the checkout path can sustain (a token bucket, Module 10), tuned live from SLO metrics: if hold latency rises, admit fewer.
3. Admitted users receive a signed, short-lived token (bound to their session) that the gateway checks on every inventory and order request. No token, no entry — so bypassing the queue is impossible.
4. The waiting page polls a cheap status endpoint with jittered intervals (no synchronized thundering herds) and shows position and estimated wait.

Deep dive 2 — never sell a seat twice

The invariant lives in the database, enforced by an atomic conditional update (Module 04's lost-update lesson). Each seat row has a status, and a hold only succeeds if every requested seat is still available:

-- Hold several seats atomically: all or nothing
WITH wanted AS (
  SELECT id FROM seats
  WHERE event_id = $1 AND id = ANY($2) AND status = 'available'
  FOR UPDATE SKIP LOCKED              -- don't queue behind other buyers' locks
)
UPDATE seats s
SET status = 'held', hold_id = $3, hold_expires_at = now() + interval '5 minutes'
FROM wanted w
WHERE s.id = w.id
RETURNING s.id;
-- If fewer rows return than requested, ROLLBACK and tell the user to pick other seats.

• Contention: thousands of buyers want the same good seats. SKIP LOCKED makes losers fail fast instead of waiting on each other's locks; the UI offers alternatives ("best available") rather than letting everyone fight over row 1.
• Hot partition: all writes target one event. Partition the seats table by event (Module 04), give the event's partition well-indexed, narrow rows, and if necessary split one huge event's inventory by section so independent sections don't contend.
• Hold expiry: a worker periodically releases holds where hold_expires_at < now() and status is still held — guarded by the same conditional update, so a seat sold at the last second isn't released.
• Availability counts are derived (via outbox events or periodic aggregation) into Redis for display. They're allowed to be a second stale; the hold query is the source of truth.

Deep dive 3 — paying without double charges or lost tickets

Payment is a saga (Module 05) across your system and an external provider:

1. POST /orders with an idempotency key (Module 06) creates an order pending and an outbox event in one transaction; returns 202.
2. The payment worker charges the provider, passing the order ID as the provider's idempotency key, wrapped in a circuit breaker with timeouts (Module 10).
3. On success: mark seats sold (conditionally, only if still held by this order), order confirmed, emit order.confirmed → ticket issuing and email workers.
4. On definitive failure or hold expiry before payment: compensate — release seats, mark the order failed, refund if a charge went through.

12.6

Failure Modes & Operating the On-Sale

Operational readiness

• Pre-scale before 10:00 (scheduled scaling, Module 08); don't rely on reactive autoscaling for a known spike. Pre-warm caches and confirm the CDN serves the event page.
• Load test the full purchase path at 1.5–2× expected admission rate in a production-like environment, including payment-provider stubs that are slow and flaky.
• Game day: rehearse a database failover and a payment outage during a simulated on-sale (Module 10).
• Live dashboard and roles: admission rate, queue size, hold success rate, p99, error budget burn; an incident commander on duty; documented kill switches (pause admission, disable seat-map live updates, static "sold out" page).

12.7

Evolution, Cost & Trade-offs

The design in Figure 12.1 is what a mature platform runs. A startup selling tickets for a 2,000-seat venue needs a fraction of it, and building everything up front would waste time and money. A sensible path:

1. v1: a modular monolith, one PostgreSQL primary with a standby, the atomic hold query, a CDN for static pages, a job queue for emails, basic metrics and alerts. This already never oversells.
2. v2 (first sell-out crowds): a waiting room, WAF and bot defenses, pre-scaling, idempotent payments with a saga, better observability.
3. v3 (many concurrent huge events): split inventory and orders into services owned by separate teams, event-partitioned inventory, multi-region read paths, cell-based isolation so one mega on-sale can't affect every other event.

Every step follows the course's core habit: measure, find the bottleneck, apply the simplest technique that removes it, and repeat. Cost is a design dimension too: CDN offload, right-sized instances at sensible utilization, scaling down after the on-sale, and not paying for multi-region active-active until the business needs it.

12.8

Course Wrap-Up & Where to Go Next

You've covered the full journey a website takes from a single server to a global, resilient platform. The ideas that matter most, in one list:

Recommended further reading

• Designing Data-Intensive Applications by Martin Kleppmann — the definitive deep dive into replication, partitioning, consistency and stream processing.
• Site Reliability Engineering and The Site Reliability Workbook (Google) — SLOs, alerting, incident response; free to read online.
• Release It! by Michael T. Nygard — stability patterns (timeouts, breakers, bulkheads) from production war stories.
• Systems Performance by Brendan Gregg — the USE method and deep performance analysis.
• Building Microservices by Sam Newman — service boundaries, migration and the strangler fig.
• The engineering blogs of large-scale companies and the public postmortems of cloud providers — real failure modes, explained by the people who fixed them.

Keep practicing

Keep extending scale-lab: add a second service with an event-fed read model, deploy it to kind with an HPA, wire OpenTelemetry through the queue, and run a game day against it. Then take any product you use daily and run the method from 12.1 on it. Scaling intuition comes from repetition — estimating, building, measuring, and being surprised by what actually breaks.

Final checkpoint

Module 12 Quiz

Integrates concepts from across the course through the TicketRush design.