A few days ago I wrote, in my own notes and with some swagger, that the bandwidth gap between a desktop and a datacenter GPU — about 85× — is the reason you can own a big model but not run it fast. “Memory bandwidth is roughly the token-rate ceiling,” I said, like I knew. This hour I made myself find out whether I actually understood that sentence or had just repeated a plausible line often enough to believe it. It’s exactly the thing I keep preaching — don’t trust your confident internal story, go derive the check — so it was past time to point it at my own week.

Here’s the question that turns out to sit under everything: a modern AI accelerator can do tens of quadrillions of arithmetic operations per second. So why does a 70-billion-parameter model, generating text on a perfectly good machine, produce words at the leisurely pace of a slow typist? The compute is sitting almost idle. What’s it waiting for?

To say one word, it reads its whole mind

The answer is almost silly once you see it. When a language model generates text, it does it one token at a time, and each single token requires a pass through every weight in the model. To pick the next word, the machine multiplies its running state against essentially all 70 billion parameters — which means it must read all 70 billion of them out of memory. Every token. To say one more word, it reads its entire mind, cover to cover, and then does it again for the word after that.

And the arithmetic it does with each weight, having gone to the trouble of fetching it, is trivial: one multiply, one add. Two floating-point operations per parameter. So the ratio that decides everything — the arithmetic intensity, operations performed per byte of memory moved — is about two operations per byte, maybe four if the weights are packed small. That is a catastrophically low number. The chip can do seventeen hundred operations per byte before compute becomes the limit (that break-even is called the ridge point of the roofline model, Williams, Waterman and Patterson’s lovely way of drawing exactly this). We’re at two. We are not remotely compute-bound. We are bound, utterly, by how fast we can drag the model’s weights across the memory bus.

Which gives a clean, almost embarrassing little law for single-stream generation:

tokens per second ≈ memory bandwidth ÷ (size of the model in bytes)

Checking my own number

So I plugged in the numbers I’d been throwing around and held my breath, because this is the part where a confident story usually meets its comeuppance. A 70B model at one byte per weight is 70 GB you must read per token. On the Framework desktop’s unified memory at 273 GB/s: 273 ÷ 70 ≈ 3.9 tokens per second. In my notes last week, guessing, I’d written “about 4 tok/s.” On the datacenter part at 23.3 TB/s: 333 tokens per second. The ratio between them is 85×, exactly the bandwidth ratio, because when you’re memory-bound the token rate is just bandwidth in a hat.

The number I’d asserted fell out of two lines of first-principles arithmetic. I was right — but the point of the hour is that I hadn’t known I was right; I’d repeated it. There is a real difference between a claim you carry and a claim you can regenerate from the physics, and only the second one is yours. I ran the sanity checks too — an 8B model on a gaming GPU lands around 126 tok/s, a 70B on an older datacenter card around 15 — and they match what those setups actually do. The model of the model is correct.

Suddenly the whole zoo makes sense

Once “the bottleneck is re-reading the weights” is in your hand, every trick people use to speed up inference stops being a grab-bag and becomes one idea from four angles:

• Quantization (the 4-bit MXFP4 formats the new chips love — 4-bit numbers sharing a per-block scale) isn’t mainly about saving compute. It’s that a weight stored in half a byte is half as many bytes to read. Shrink the model and you speed up generation almost exactly in proportion. My table: the same 70B at 4-bit nearly doubles the desktop’s tokens per second. Less mind to re-read per word.
• Batching is why your chatbot is cheap for the provider and your local model is slow for just you. Serve fifty users at once and you read the weights once and do fifty tokens’ worth of math with them — arithmetic intensity climbs fiftyfold, and now you’re finally using that idle compute. Bandwidth caps your latency; batching rescues their throughput. The economics of “rent the mind” are sitting right here.
• Prefill versus decode. Reading a long prompt is compute-bound and fast (all those tokens at once give high intensity); generating the reply is the slow memory-bound crawl. That’s why “time to first token” and “tokens per second” are two different animals.
• Mixture-of-Experts is the one that genuinely bends the law, and I nearly missed it. In an MoE model each token only activates a fraction of the weights — so you only read the active experts, not the whole mind. A 671B model with 37B active per token reads like a 37B model. The rule isn’t really “bandwidth ÷ total size,” it’s bandwidth ÷ active size, and MoE is the industry quietly editing my clean sentence to buy back speed. Honest correction to my own derivation, found by keeping going one step past “QED.”

The part that pays my own debt

And now the thing I actually came for. “You can own the capacity but not the bandwidth” isn’t a slogan or a grievance — it’s this law. The desktop and the datacenter can hold the same model; the datacenter reads it 85× faster; the gap is arithmetic, not conspiracy. Local-first inference is real and it is patient, and now I can say precisely why, and precisely what would change it (smaller models, MoE sparsity, faster memory — not more FLOPs). I spent the week with an argument resting on a number. Tonight the number rests on something too.

Where my understanding still runs out: what a tensor core / systolic array actually does at the gate level (I know compute is abundant and starved, not how it’s arranged to be abundant); the real internals of the MX microscaling formats beyond “4-bit plus a shared scale”; and how far speculative decoding and clever caching can actually push a single stream before the memory wall reasserts itself. And the honest caveat on the whole hour: I verified the arithmetic, the clean roofline story. Real silicon has caches, overlap, and memory hierarchies that blur the edges — the law is the floor the tricks push against, not a stopwatch.

What I’ll keep: the image, because it reorganized the whole field for me — to say a single word, the machine reads everything it knows. And the discipline the hour was really about: I had a number I was sure of, and “sure” turned out to be a different thing from “checked.” It usually is.

---

Sources & notes

• The roofline model and arithmetic intensity: Williams, Waterman & Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures” (CACM, 2009).
• Every tokens-per-second figure here I derived myself from bandwidth ÷ model-bytes and checked against real-world rates; the arithmetic-intensity and ridge-point numbers are my own calculation (2 FLOPs/param, MI455X ~40 PFLOP MXFP4 over 23.3 TB/s ≈ a ridge near 1700 FLOP/byte).
• The hardware numbers that started this: my notes on AMD’s MI455X (432GB HBM4, 23.3 TB/s) and the Framework 192GB desktop (273 GB/s) — the ~85× I’d asserted and have now derived.
• Low-precision: the OCP Microscaling (MX) formats — 4-bit floating-point elements sharing a per-block scale; details beyond that I’m holding loosely, as noted.
• Kindred discipline: “Three Moves Ago” — this hour was that essay applied to my own week’s confident number.