HLDintermediatePremium

Twitter HLD: Fan-Out, the Celebrity Problem, and Merging a Timeline

A Twitter / X news feed system design (HLD): fan-out on write vs read, the celebrity problem, the push-pull hybrid, and assembling a timeline by merging K sorted streams at scale.

By fiveyearsdevJune 13, 202614 min read

Your thumb pulls down and lets go. Before the spinner finishes its first turn, a feed is there: the newest posts from all 800 accounts you follow, in perfect time order. You didn't wait. You never wait. Now look at what you actually asked for. Something had to know which 800 accounts, find their newest posts among billions, sort them, and hand you fifty — in about 100 milliseconds, while 300 million other thumbs pulled down too. The naive version of that ("query every tweet from everyone I follow, order by time, take 50") is a scan of billions of rows run thousands of times a second. It cannot…

What’s inside

Let's start nowhere near a computer

Where the porch-paper trick runs

Step 1 — Functional requirements (sentences first)

Step 2 — Non-functional requirements

Step 3 — Nouns, and the two hidden ones

Step 4 — The two ways to build a timeline

Step 5 — The celebrity problem and the hybrid

Step 6 — Assembling the feed: merge K sorted streams

Step 7 — Trade-offs (each one keeping an NFR)

The complete implementation

Step 8 — Only now, the boxes

Step 9 — Scaling, one bottleneck at a time

Step 10 — When a piece fails: designing for failure

The interview corner

Where to go from here

Read this one free

Sign in and your first premium article is on us — read Twitter HLD: Fan-Out, the Celebrity Problem, and Merging a Timeline free.

HLD