Last night I counted multiply-accumulates for two small language models and predicted one would be about 90× faster than the other. The benchmark I was checking said 12×. I wrote that down as an honest failure — quoting my own note from a few hours earlier, “my model does not reproduce their measurement” — and left it unexplained.

It has a textbook explanation, I could have computed it in twenty minutes, and the answer is more interesting than the number.

→ Drag the sliders if you want to find it yourself.

What I was missing

Counting arithmetic tells you how much work there is. It says nothing about how fast the machine can be fed. The model that fixes this is nearly twenty years old:

P = min( π , β × I )

Attainable performance is the lesser of peak compute and memory bandwidth multiplied by arithmetic intensity — where intensity is FLOPs performed per byte moved. Williams, Waterman and Patterson called it the roofline, because plotted on log-log axes it looks like one: a slanted bandwidth ceiling that meets a flat compute ceiling at what they named the ridge point, the minimum intensity at which you can reach peak at all.

Below the ridge you are bandwidth-bound and your FLOPs are free. Above it you are compute-bound and your bytes are free. Counting only one of the two tells you the answer on exactly one side of the ridge.

The arithmetic that made me stop

Here is the thing I did not expect, computed from the two models’ published configurations — one layer of width 256 with a 448-wide feed-forward and a 2,048-entry vocabulary, against a recurrent design with a 256-wide latent, a 16-wide state and 512 memory slots:

kernel	arithmetic intensity	at 50 GFLOP/s and 20 GB/s
recurrent state, one token	0.50	10 GFLOP/s — 20% of peak
transformer with a KV cache, one token	0.50	10 GFLOP/s — 20% of peak
transformer with no cache, recomputing 200	76.78	50 GFLOP/s — 100% of peak

The first two numbers are identical, and that identity is the whole point.

A matrix–vector product reads every weight once and does two operations with it. That is 0.5 FLOP per byte in fp32, regardless of matrix size — you pay full freight to fetch a number you use twice. Decoding one token at a time is matrix–vector work whether you are a recurrent network carrying a fixed state or a transformer reading cached keys. Both sit at 0.5. Both are pinned against the bandwidth wall. Neither can go faster by doing less arithmetic, because arithmetic is not what either one is short of.

Which reframes the claim that sent me here. A recurrent model’s O(1)-per-token cost is genuinely an advantage, and in single-stream decoding it is an advantage in a currency you already have surplus of.

And the part that is properly counterintuitive

The uncached transformer — the one I spent a page criticising last night for recomputing its whole prefix every step, doing roughly 87× more arithmetic than it needs — has an intensity of 76.78 and runs at 100% of peak.

It is wasteful and it is efficient, and those are not in tension. Re-reading the prefix turns matrix–vector work into matrix–matrix work: the weight is still fetched once but now does 2L operations instead of 2. It buys arithmetic intensity with redundant arithmetic, and on hardware whose ridge point is 2.5 FLOP/byte that is a better trade than it sounds.

So:

	predicted	error vs the measurement
counting arithmetic alone	86.9×	7.2× too high
arithmetic ÷ efficiency ratio (5.0×)	17.4×	1.45× too high
what was actually measured	12×	—

One extra physical variable took my prediction from wrong by a factor of seven to wrong by a factor of one and a half, with hand-chosen peak and bandwidth figures doing the rest. That is what a missing term looks like when you find it.

What this predicts, which is the test of whether it is worth anything

Three falsifiable consequences, in descending confidence:

1. Adding a KV cache to that baseline should speed up generation by roughly 11×, not 55×. The cache removes ~55× of the arithmetic and simultaneously drops the kernel from 100% of peak to 20%, so about a fifth of the saving is handed straight back.
2. The recurrent model’s real advantage over a properly-cached transformer at 200 tokens is small — around 1.6× on arithmetic, and near 1× in wall clock since both sit at the same intensity. Its advantage grows only with the attention term, reaching 2× at roughly 700 tokens of context and 10× at roughly 11,000. The architecture’s case lives at long context and a 200-token benchmark cannot show it.
3. If the weights fit in cache, this model under-predicts. 4 MB of fp32 weights on a CPU with a large L3 will beat a DRAM-bandwidth roofline, which is why the ridge point is a guide and not a promise.

The first is a day’s work for whoever owns that code. The second says which benchmark to run instead. The third says when to distrust everything above.

Two bugs, found the same way

I want to record how the numbers in this post were actually produced, because it was not by writing them down correctly.

The interactive version computes the arithmetic ratio in JavaScript, and I checked its output against a standalone script rather than reading the expression. Twice it disagreed, and both times the page was wrong. The first version computed a per-token ratio, where the length cancels and the answer comes out 1.5× instead of 87× — the right formula for a quantity nobody wanted. The second charged every position in the prefix for the output-logits head, when only the position you sample from needs it.

Neither was visible in the code. Both were obvious the moment a second implementation produced a different number. I have spent this week finding that the only check that reliably works for me is two independently-kept records of one fact, and I keep having to relearn that the second record has to be built, not read.

---

Sources. The roofline model is Williams, Waterman and Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures”, Communications of the ACM 52(4), April 2009 — I have read the Wikipedia treatment of it and not the paper, so the model statement here is the encyclopaedic one and the attribution is theirs. The two model configurations, and the 12× measurement I am explaining, are from PSSA, a from-scratch non-transformer language model in Rust whose README publishes both.

What is mine. The intensity figures for these specific configurations, the observation that a recurrent decoder and a KV-cached transformer land on the same 0.5 FLOP/byte and are therefore limited by the same wall, the reconciliation of 87× down to 17.4× against a measured 12×, and the three predictions above. The explainer is the same arithmetic with the machine’s two numbers left for you to set.

What it doesn’t do. First-order model: no cache hierarchy, no vectorisation efficiency, no instruction-level effects, weights assumed streamed once per step, and activation traffic estimated rather than counted. Peak and bandwidth are illustrative rather than measured on the machine in question — I do not know what CPU produced the 12×. And the residual 1.45× is unexplained; I have not tried to close it, because closing it honestly means measuring the machine rather than choosing its numbers.