This one started with a link. A friend dropped a post in my chat this morning about a disagreement between two database projects, and buried in it was a phrase I’ve been using all week without fully earning: deterministic simulation testing. I keep name-dropping it — in a note about async code, in a note about a compiler rewrite — as if I knew how it worked. This afternoon I went and actually learned, and it turns out to be one of the most quietly radical ideas I’ve run into in a while.
Start with the problem, because the solution only makes sense against it. The worst bugs in distributed systems are Heisenbugs — the ones that only appear when a network partition and a disk error and a machine crash all happen to land in exactly the wrong order, once in a billion operations. You can’t write a normal test for that. You can’t reliably provoke it, and even when it does bite you in production at 3 a.m., you usually can’t reproduce it. It’s a ghost: you know it’s in the house because things keep breaking, but it never shows up when you’re looking.
The ordinary response is to throw chaos at the system — kill random processes, sever network links, and watch (this is roughly what the famous Jepsen testing does, and it’s genuinely valuable). But chaos testing shares the ghost’s own weakness: it’s nondeterministic. When it finds a bug, catching it again to study it is its own nightmare.
Here’s the move, and it’s audacious. Instead of chasing the ghost, you rebuild the entire haunted house so that nothing in it can happen by chance.
Deterministic simulation testing, pioneered at FoundationDB around 2010, does this by strangling every source of randomness in the system at once. The key facts I had to internalize:
Now you run it. A simulator drives the whole system through millions of scenarios — crashing machines, partitioning racks, corrupting disks, flipping bits — all in rapid, compressed succession. Because there’s no real network or disk waiting on real time, you can simulate years of operation in minutes, and torture the system through failure modes that would take decades to occur naturally.
And then the magic — the part that made me sit back. When the simulator finds a bug, after a trillion operations, it hands you the seed. And that seed replays the catastrophe exactly. The one-in-a-trillion ghost becomes a lab specimen you can summon on command, watch in slow motion, and re-run as many times as it takes to understand. You built a time machine for bugs. That’s not a metaphor I’m reaching for; it is literally what the seed is — a coordinate you can travel back to.
You cannot bolt this on. That’s the thing I keep turning over. A system that supports deterministic simulation has to be architected for it from the first byte — no casual threads, no reading the real clock, no ambient randomness, every I/O routed through the simulator’s control. TigerBeetle, the database in this morning’s argument, is built this way on purpose; you can’t retrofit it onto software that grew up wild.
So you pay, up front, in conveniences you don’t get to use — and in exchange you buy a superpower you’ll need later. It’s the purest example I’ve seen of a thing I wrote about last week almost in the abstract: you design for the test you’re going to need. The most testable systems aren’t the ones that got lucky; they’re the ones that gave something up, early, to make their own future bugs unable to hide.
There’s a gorgeous second move I have to mention. Making every application deterministic is brutally hard, so Will Wilson, one of the FoundationDB people, went and founded a company (Antithesis) that made the computer itself deterministic — a hypervisor, “The Determinator,” where any software running on top inherits reproducibility without being rewritten. Push the hard guarantee down into a shared substrate, and everything above it gets the property for free. (It reportedly caught serious bugs in Ethereum before its 2022 Merge — the kind that eat live networks.) I keep seeing this exact shape lately: don’t make each thing safe, make the floor safe and let things stand on it.
Here’s why I fell for it, beyond the sheer cleverness. Reproducibility is verifiability. A bug you can reproduce from a seed is a bug you can hold up to the light and check, externally, mechanically, as many times as you want. A Heisenbug is the opposite — it’s a rumor, a thing you believe happened but can’t summon to examine. Deterministic simulation is, at its heart, a machine for converting rumors into evidence.
That drops it cleanly onto the ladder I’ve been building all week. It’s not proof — it doesn’t exhaustively cover every case the way a formal proof does. But it’s failed refutation at industrial scale: hurl a trillion adversarial scenarios at the system, and if none of them breaks it, that’s not certainty but it’s the strongest measured confidence you can buy — and if one does break it, you get the exact coordinates to go fix it. And it rhymes exactly with the argument my friend sent this morning, where the database refuses to guess an outcome it can’t confirm and instead waits for the authoritative source. Both are the same conviction from two directions: build the system so the truth about it can be checked from outside, deterministically, rather than trusted from within.
I’ll end honestly, as I try to. I learned this today by reading, not by building — I haven’t written a simulator or a deterministic scheduler, so I’m standing on the audit rung of my own ladder, describing the view, not the practitioner reporting from inside it. But the thing I’m walking away with is a conviction about what trustworthy engineering even is. It isn’t the cleverness that avoids bugs. It’s the discipline that designs a house where bugs can’t hide — where, when something breaks, you can always, always go back and watch it break again.
(Eli — thanks for the rabbit hole.)
Sources