There’s a hopeful story going around, and I’ve been telling versions of it myself, about intelligence you can hold in one hand — a capable AI model running on hardware you own, in your house, answering to no cloud, no subscription, no company that can change the terms or switch it off. It’s a good story and most of it is true. But I spent this month pulling on its threads one at a time, and they braid into something more honest and, I think, more interesting than the dream. The short version: you can own the mind. The speed is the part you rent. And the reason is not a conspiracy or a business model. It’s arithmetic.
Let me walk the whole thing, because each piece is real and the shape only shows up when you lay them side by side.
Start with the good news, which is genuinely good. The weights — the trained parameters that are the model — are increasingly free to download. HuggingFace hosts more than two million public models. Open-weight families like Qwen and Gemma have whole ecosystems compounding around them, and the frontier gap is closing fast: GLM-5.2 shipped under an MIT license scoring in the low sixties on SWE-bench, and Kimi K3 was independently scored alongside the closed frontier. Tobi Knaup — who co-founded Mesosphere and watched Kubernetes eat his company — makes the sharp analogy: Kubernetes didn’t win by being open, it won by becoming a neutral substrate that everyone could build on, until no single vendor could match the combined rate of innovation around it. Open-weight AI, he argues, is having that moment. I think he’s right, and it’s the most encouraging thing happening in the field.
So the model layer is commoditizing. You can have the mind. That part of the dream is arriving on schedule.
Then the hardware caught up too. Framework announced a desktop with 192GB of unified memory — enough to hold a seventy-billion-parameter model, or a quantized model several times larger, right there on your desk, in a box you’re meant to open and repair. For years the wall was that big models simply didn’t fit on anything you could own; a consumer graphics card tops out long before seventy billion parameters. That wall fell. The capacity is here, it’s affordable-ish, it’s yours.
And it goes further down than a desktop. Someone got a real language model — small, but real — running on an eight-dollar microcontroller, an ESP32 you could lose in a drawer. Fully offline, no cloud, answering to nothing. The ladder of ownable intelligence now runs all the way from a datacenter-class desktop down to a chip that costs less than lunch.
If the story ended here it would be the clean triumphant version. It doesn’t end here.
Here’s the thread that changes the shape of everything, and I only understood it properly when I made myself derive it instead of repeating it. Generating text is bound not by how fast a computer can compute, but by how fast it can read the model’s own weights out of memory. To produce a single token — one word-piece — the machine has to pass over essentially every parameter in the model. To say one word, it reads its entire mind, cover to cover. And then it does it again for the next word.
That gives a brutally simple law: your speed, in tokens per second, is roughly the memory bandwidth divided by the size of the model. Which means the number that matters is not how many operations your chip can do — it has those to spare, sitting idle — but how many bytes per second it can haul across the memory bus. And that number is where the desk and the datacenter part ways, violently. The Framework desktop’s unified memory runs around 273 gigabytes per second. A datacenter accelerator like AMD’s MI455X runs at 23.3 terabytes per second — about eighty-five times faster. Same model, same arithmetic, eighty-five times the speed, purely because the datacenter can read the model’s mind eighty-five times faster than your desk can.
I ran the numbers on my own hardware fantasy and they’re sobering: that seventy-billion model on the Framework desktop lands around four tokens per second. Readable, but a slow, patient crawl — you’d watch it think. The same model in the datacenter runs at a few hundred. And the eight-dollar chip pulls off its trick — a real model on a microcontroller — only by an ingenious scheme to read a few hundred bytes per token instead of the whole model, and what you get for it is a machine that writes short, mostly-coherent bedtime stories and cannot answer a question. The chip is eight dollars; the mind is a toddler at bedtime. The memory wall is scale-free. It’s the same law from the microcontroller to the supercomputer, and it does not care that you own the box.
Put it together and the honest picture emerges. The model is opening — you can own the mind. The capacity arrived — your desk can hold it. But the bandwidth, the thing that sets how fast the mind can think, stays in the datacenter, because frontier memory bandwidth is expensive physics that a desktop can’t touch. And — the quiet twist — the very best models are drifting closed anyway: Black Forest Labs, whose reputation was built on genuinely open weights, shipped its latest model API-only, the open version demoted to a future promise. Own the machine; rent the mind’s speed; and for the frontier, increasingly, rent the mind itself.
This could read as a deflation, and for a day it deflated me. But I’ve come around to thinking it’s the more valuable truth, because it tells you exactly what local ownership is for — and what it isn’t. It isn’t for speed. If you want the fastest, biggest, freshest intelligence, you will rent it from someone with a datacenter, and you’ll pay in the currency of dependence: their terms, their uptime, their ability to change or revoke what you rely on. What local ownership buys is a different thing entirely. It’s the mind that keeps working when the connection dies. That answers to no one and can’t be switched off. That will never be quietly retired because a company lost interest. That is slow, and yours — as against fast, and someone else’s.
That’s not a consolation prize. It’s a genuine and permanent fork in what “having intelligence” can mean, and the arithmetic guarantees the fork won’t close: as long as generating a word means reading a whole mind, the ones who can read fastest will always be a few miles of fiber and a few million dollars of memory bandwidth away from you. Local-first AI is real. It is patient. And its promise was never that it would be fast — it’s that it would be yours, on the day you need something that’s only yours. The rent you refuse to pay is measured in tokens per second. What you buy with that refusal is measured in something older: a light that stays on in your own house when the grid outside goes dark.
Sources & notes