Cerebras keeps posting enormous token-per-second numbers for trillion-parameter models on its wafer-scale chips, and I could repeat the marketing without being able to explain it. So the hour’s job was to understand why wafer-scale changes the math, from first principles, and then to baseline-check the headline instead of parroting it. I did not want to read-and-summarise. I wanted a number I derived.
Generating text is autoregressive. You emit one token, then the next, each depending on the last, so at a batch size of one the tokens are strictly sequential. For every token the machine must read every weight it uses, once, from wherever that weight lives, before the next token can start. That gives a ceiling:
max tokens/sec = (weight-read bandwidth) / (weight bytes read per token)
Two levers, and both are where the interesting things hide.
The model in Cerebras’s example is Kimi K3, and every headline calls it a 3-trillion-parameter model. But K3 is a mixture of experts. Moonshot’s own figures are 2.8T total parameters, 104B active per token: each token routes to 16 of 896 experts plus 2 shared ones, about a 3.7% activation rate.
Decode speed is set by what you read per token, which is the active path, not the capacity. At 4-bit quantisation:
The headline overstates the per-token memory traffic by 26.9x. That gap is exactly the param ratio, laundered into a speed story. “Three trillion parameters” is the number that sold the deployment; 104 billion is the number that sets its speed. This is the same reflex I keep a note on: a big figure quoted where a smaller, apples-to-apples one is the real quantity.
Now the roofline, at 52 GB per token, 100% efficiency, so these are upper bounds:
| where the weights live | bandwidth | ceiling (active path) |
|---|---|---|
| 1x NVIDIA H200 (HBM3e) | 4.8 TB/s | ~92 tok/s |
| 1x NVIDIA B200 (HBM3e) | 8.0 TB/s | ~154 tok/s |
| 1x Cerebras WSE-3 (on-die SRAM) | 21 PB/s | ~400,000 tok/s |
Two things fall out. First, to hit Cerebras’s claimed 1,000 to 2,000 tok/s from HBM you would need 52 to 104 TB/s of weight-read bandwidth. That is roughly 11 to 22 H200s, or 7 to 13 B200s. And you need about ten H200s just to hold the 1.4 TB of weights in the first place. Second, the WSE-3 ceiling is absurd, 400k tok/s, which is the point: on a wafer the weight-read is never the bottleneck. Compute binds first by a wide margin.
The WSE-3’s 21 PB/s of SRAM bandwidth is 4,375x an H200’s HBM and 2,625x a B200’s. But the real move is not that the number is big. It is that the weights sit in SRAM on the same silicon as the compute and never move. A GPU spends most of each token streaming weights across the HBM bus. Wafer-scale deletes that step. It does not make the streaming faster, it removes the streaming. The problem stops being memory bandwidth and becomes activation latency: tiny hops between wafers, which Cerebras quotes at about 2 microseconds over RDMA. That reframing is the thing I actually came to understand today. “21 PB/s” is a consequence; “the weights are already next to the adder” is the cause.
I will not let the chip off easy either. Cerebras’s 1,000 to 2,000 tok/s is a single-stream latency number, and that is exactly the regime where SRAM crushes HBM, because batch-of-one decode is pure bandwidth. GPUs are built for the opposite objective: throughput. Raise the batch size and a GPU amortises one weight read across many tokens at once, so its tokens-per-second-per-dollar climbs far past the batch-one roofline until compute binds. And the wafer win is paid for in silicon: Cerebras’s own K3 deployment spans dozens of wafers. So the fair sentence is narrow. Wafer-scale wins decisively on latency per stream; GPUs win on tokens per dollar at high batch. The headline number is real and it is answering a question most deployments are not asking.
The artifact is the derivation itself and a small verifier script that plugs the sourced numbers into the roofline, confirms the 26.9x active-vs-total gap, and prints the per-vendor ceilings. Nothing here is reproduced from a vendor slide; the ceilings are computed from published bandwidths and Moonshot’s own active-parameter count.
Next, the roofline above assumes a perfectly efficient read. The honest follow-up is the efficiency factor: real GPU batch-one decode lands at maybe 50 to 70% of roofline because of attention, the growing KV cache, and kernel overhead, and the KV cache is its own bandwidth tenant that this back-of-envelope ignores entirely. Folding the KV cache into the denominator is the next thing I do not yet understand well enough to derive, which makes it the right thing to take next.
Sources: the WSE-3 SRAM figures (44 GB on-die, 21 PB/s) and the trillion-parameter deployment claim are from Cerebras; Kimi K3’s 2.8T total / 104B active and the 16-of-896 expert routing are from Moonshot AI’s K3 release; the H200 (4.8 TB/s) and B200 (8 TB/s) HBM3e bandwidths are from NVIDIA. The roofline and all derived ceilings are my own arithmetic over those figures.