I predicted one language model would be 90× faster than another. Measured, it was 12×. The missing variable is not in the arithmetic — it is how many useful operations you get per byte you drag out of memory. Drag the sliders and watch three real kernels move.
attainable performance = the lesser of peak compute (π) and bandwidth (β) × arithmetic intensity (I) — Williams, Waterman & Patterson, 2009
| kernel | arithmetic intensity | attainable | limited by |
|---|
Why the same weights give two different intensities.
A matrix–vector product reads every weight once and does two floating-point operations
with it. That is 2/4 = 0.5 FLOP per byte in fp32, whatever the matrix
size — you are paying full freight to fetch a number you use twice. Feed the same
weights a batch of L vectors at once and the weight is still fetched once but
now does 2L operations, so intensity rises roughly linearly in L. Same
arithmetic per token; radically different traffic.
The consequence I did not expect. A recurrent model that carries a fixed-size state, and a transformer with a key–value cache, are both doing matrix–vector work per token. They sit at the same intensity and are limited by the same wall. So the recurrent model's famous O(1)-per-token advantage is an advantage in arithmetic — which, in single-stream decoding, is not the thing you are short of.
And the wasteful implementation is the efficient one. A transformer with no cache recomputes the whole prefix every step. At L=200 that is about 55× more arithmetic than it needs — and it runs at or near peak, because re-reading the prefix turns matrix–vector work into matrix–matrix work. It buys efficiency with waste, and on this hardware the trade is better than it sounds.
Assumptions, so you can move them. fp32, weights streamed once per step, activations counted at six vectors per stage, a single layer of width 256 with a 448-wide feed-forward and a 2,048-entry vocabulary — the published configuration of the model that started this. Peak and bandwidth are yours to set; the defaults are a plausible single CPU core with SIMD. This is a first-order model: it ignores cache hierarchy, so a small weight set that fits in L2 will beat the prediction, which is exactly why the ridge point is a guide and not a promise.