catalogue / apis
APIs & the network
Latency is a tax on conversations, not on bytes.
8 live panels on this page
When the server gets busy
How long a request takes as traffic rises, for three database connection pool sizes. Each line sits near 20 ms while there is headroom, then goes vertical, but how much headroom depends on the size of the pool: the pool of 2 is already at 56 ms when 80% busy, while the pool of 32 is still at 29 ms at 95%. The dashed line is a 250 ms target.
Assumes work averaging 20 ms per request (varying randomly), randomly timed arrivals and a queue that never turns anyone away
work 20 ms pools 2 8 32 SLA 250 ms
- one line per connection pool; the number on each wall is its size
| Connection pool | Max throughput | Busy | Response time | Status |
|---|---|---|---|---|
| 2 connections | 100/s | 10% | 20.2 ms | within SLA |
| 8 connections | 400/s | 3% | 20 ms | within SLA |
| 32 connections | 1,600/s | 1% | 20 ms | within SLA |
12 API calls to draw one screen
A screen needs 12 pieces of data from a server some distance away. Dots are requests heading out, streaks are responses coming back (longer streak, bigger payload), and the timeline below lays out every round trip. Asking once per piece pays the trip across the wire 12 times in a row; one batched call pays it once.
Assumes light in fibre at 200,000 km/s over a route 1.5× the straight-line distance, plus 0.5 ms of switching per round trip; 2 ms of server work per call; 2 KB per item over a shared 50 Mbit/s link; warm keep-alive connections; at most 6 connections per host (HTTP/1.1)
calls 12 connections 6 server 2 ms item 2 KB link 50 Mbit/s
- linear: one more round trip per call
- step: one more per 6 calls
- flat: one, however many pieces
- faint bar: server work and download; solid: on the wire
| Approach | Time to render | Status | Calls | Round trips waited | Data |
|---|---|---|---|---|---|
| Chatty | 1.04 s | abandoned | 12 | 12 | 24 KB |
| Parallel | 176 ms | noticeable | 12 | 2 | 24 KB |
| Batched | 89.9 ms | instant | 1 | 1 | 24 KB |
At 5,570 km every round trip costs 84 ms, so the chatty screen spends 1.01 s just waiting on the wire and renders 11.5× later than the batched one. Parallel calls cut the 12 waits to 2 because the browser opens at most 6 connections per host over HTTP/1.1; over HTTP/2 all 12 share one connection and land close to the batched time, but the server still handles 12 requests. An 80 ms round trip is about 5,300 km in this model, a little short of London to New York.
A fast average, and users stuck waiting
Requests to one endpoint pile into a histogram of how long each took. The mean (150 ms) and median (58.2 ms) sit well inside a 300 ms SLA, yet the slowest 1% take 3 s or more, and that 1% is not 1% of your users: a page that makes 50 requests hits at least one of them on 39% of page views.
Latency mixes three log-normals: 85% cache hits (median 55 ms), 13% database queries (median 220 ms), 2% stalls from GC pauses, lock waits and cold caches (median 3 s). A page view waits for all its requests, each drawn independently.
SLA 300 ms database 13 % stalls 2 % stall 3,000 ms run 3,000 users 60
- requests faster than the p99
- the slowest 1%
- median page view at 50 requests
60 users loading pages. A dot turns rose when a page view waited on a request slower than p99 (3 s). - of the last 0 page views (model: 1 - 0.99^50 = 39%)
| Statistic | So far | Model | 300 ms SLA |
|---|---|---|---|
| mean | needs 1 | 150 ms | |
| p50 | needs 2 | 58.2 ms | |
| p90 | needs 10 | 199 ms | |
| p99 | needs 100 | 3 s | |
| p99.9 | needs 1,000 | 8.05 s | |
| median page view | - | 2.23 s | over SLA |
One page, many backend calls
A page that calls services in parallel can only render when the slowest one answers. Every service here is healthy, fast 99 times in 100, yet the chance a page hits at least one slow call is 1 − 0.99ⁿ: 11% of pages at 12 calls, 63% at 100, and from 69 calls the typical page is slow.
Assumes independent calls: 99% take 10 ms plus a random wait averaging 15 ms (at most 150 ms), 1.0% take 800 ms plus one averaging 200 ms. A page is slow past 500 ms.
slow 1 % slow call 800 ms hedge at p95
- plain: wait for every call
- hedged: backup call after the p95
- share of the last 1,000 simulated pages
| Calls | Pages slow | Page p50 | Page p99 | Extra calls |
|---|---|---|---|---|
| Plain calls | 11% | 56 ms | 1.3 s | 0% |
| Hedged calls | 0.12% | 56 ms | 109 ms | +5.0% |
Hedging: once a call has run past the service's own p95 (58.1 ms), send the same request to another replica and take whichever answers first. It costs 5.0% more calls and here cuts slow pages from 11% to 0.12%, but only because this model makes every call slow independently. When the whole service is slow (a bad deploy, a saturated database) the backup is slow too, and hedging just adds load at the worst moment.
When every client retries at once
Two identical fleets of clients send 150 requests a second to a service that handles 200, and for a moment it slows to 10% of its speed. Each chart stacks retries on top of the normal traffic against the dashed capacity line, with calls actually answered as the solid line: clients that retry straight away can keep the service buried long after the blip is over, while clients that back off at random and stop calling when failures pile up are back within seconds.
Assumes a 1 s client timeout, a server queue of 400 that keeps working on requests their client has given up on, 3 retries per call, and a blip at 10% capacity; a real run of every request in 10 ms steps, the same seeded traffic for both fleets
capacity 200 /s traffic 75 % timeout 1 s queue 400 retries 3 blip at 10 %
- first attempts
- retries, sent at once
- retries, after backoff
- capacity
- calls answered in time
| Clients | Calls answered | Retries sent | Wasted work | Since the blip | Status |
|---|---|---|---|---|---|
| Retry at once | 16% | 5,159 | 2,018 requests | down 10.0 s after it | outage |
| Backoff, jitter and a breaker | 79% | 46 | 255 requests | back 1.5 s after it | healthy |
After a 2.0 s blip, the clients that retry at once are still down 10.0 s after it ended, and would stay down: every request the server finishes is one its client already gave up on. They answer 16% of calls over the run. The fleet with backoff and a breaker fails calls fast during the blip, is back 1.5 s after it, and answers 79%. Wasted work is requests the server finished after their client had stopped waiting.
An API that expands everything it is asked to
One customer fetched with ?expand= and no depth limit. Every dot is an object in the JSON response and every row is one more key on the expand path: each level multiplies the one above it by how many children each parent has, so the tree outgrows the panel by the first row and the response goes from 228 B to 160 MB across 8 levels.
Assumes 20 orders a customer, 5 items an order, 30 reviews a product, the sample objects' real JSON sizes, serialising at 100 MB/s and sending at 100 Mbit/s (12.5 MB/s); the database queries behind each level are left out and would only add
GET /customers/42
- objects
- 1
- response size
- 228 B
- serialise + send
- 20.5 µs
orders 20 items 5 reviews 30 cap 2 link 100 Mbit/s
The orders row holds 60,000 objects: drawn at 0 px apart it would need a panel 0× this wide, so only the middle is drawn and the count on the right is what fell off.
- no depth limit: each level multiplies the size
- depth capped at 2: ids past that, size stays put
| API | Objects | Response | Serialise | Send | Total | Status |
|---|---|---|---|---|---|---|
| no depth limit | 1 | 228 B | 2.28 µs | 18.2 µs | 20.5 µs | instant |
| depth capped at 2 | 1 | 228 B | 2.28 µs | 18.2 µs | 20.5 µs | instant |
The fix is a server rule, not a client plea: cap expansion depth (return ids past level 2), page nested collections (orders?limit=10 with a next link) and let clients ask for only the fields they render (fields[product]=name,price).
Turning a response into JSON and back
An API response of orders, timed for real in this browser as you watch: JSON.stringify to build it, JSON.parse to read it, and gzip and gunzip around the wire. The timeline plays one response sent plain and sent gzipped; the chart below puts every stage against payload size. Stringify and parse run in a straight line: ten times the payload, ten times the CPU. gzip carries a fixed cost on every call as well, so on small responses it spends more CPU than the wire time it saves.
Timed here, best of 3 runs (gzip through the browser's streaming API, which costs more per call than a server's own gzip), at 1 KB to 1 MB; larger sizes are extrapolated at the 1 MB speed. Assumes a 100 Mbit/s link with no round-trip latency, stages run one after another (real servers stream and overlap them), a server that stringifies and gzips as fast as this browser, and a client that gunzips and parses as fast as it
link 100 Mbit/s server 1× client 1×
- CPU: stringify, parse, gzip + gunzip
- network at 100 Mbit/s
- extrapolated past 1 MB
- timed point
- estimate, not yet measured
| Stage | Runs on | At 1 MB | 1 KB | 1 MB | 100 MB (est.) |
|---|---|---|---|---|---|
| JSON.stringify | server CPU | 3.33 ms | 3.33 µs | 3.33 ms | 333 ms |
| gzip | server CPU | 25 ms | 25 µs | 25 ms | 2.5 s |
| wire, plain | network | 80 ms | 80 µs | 80 ms | 8 s |
| wire, gzipped | network | 12 ms | 12 µs | 12 ms | 1.2 s |
| gunzip | browser CPU | 3.33 ms | 3.33 µs | 3.33 ms | 333 ms |
| JSON.parse | browser CPU | 4 ms | 4 µs | 4 ms | 400 ms |
| total, plain | end to end | 87.3 ms | 87.3 µs | 87.3 ms | 8.73 s |
| total, gzipped | end to end | 47.7 ms | 47.7 µs | 47.7 ms | 4.77 s |
At 1 MB, stringify and parse take 7.33 ms, 8.4% of the plain response. gzip shrinks it to 150 KB, saving 68 ms on the wire for 28.3 ms of CPU: worth it here, and worth it at this size on any link slower than 240 Mbit/s. Parsing a 100 MB response would freeze the page for about 400 ms (extrapolated).
When a serverless function goes cold
A serverless function taking 120 requests an hour, simulated live. Every setting of the slider sends the same 120; only how they bunch up changes. A running instance answers in 20 ms, but one left idle for 10 min is shut down, and the next request waits 800 ms for a new one to boot. On platforms that give every in-flight request its own instance, as AWS Lambda does, a burst boots one per request, all at once. So p99 is not a curve but a step: at the same volume, it depends on how the traffic bunches up.
Simulated for real: bursts at random times, requests in a burst arrive together and each needs its own instance, 20 ms warm, 800 ms cold start, idle instances shut down after exactly 10 min. Steps: the same model solved exactly
traffic 120 /h idle timeout 10 min cold start 800 ms warm 20 ms
- cold start: boots, 820 ms
- served warm, 20 ms
- idle; its bar is the time left before it is shut down
- p99 latency
- p50 latency
- the simulation above, last 2 h
| 8 at once, every 4 min | Model | Simulated, last 2 h |
|---|---|---|
| requests that hit a cold start | 8.2% | none yet |
| p50 latency | 20 ms | none yet |
| p99 latency | 820 ms | none yet |
| average latency | 86 ms | none yet |
Sent one at a time, the 120 requests keep a single instance busy enough that it is never shut down. Bunch them into bursts of 4 and p99 is still 20 ms; bursts of 5 are only a little further apart, but now more than 1 in 100 bursts finds its instances gone, and p99 jumps to 820 ms, 41× worse, while the average moves far less (25 ms to 35 ms). Because gaps between bursts vary, p99 jumps when bursts are on average 2.17 min apart, long before the average gap reaches the 10 min timeout.