Last week the mold linker finished a rewrite from C++ to Rust and reported, in one flat sentence, that it had gained nothing:
Linking performance remains on par with 2.42.1.
The project whose entire identity is being fast changed implementation language and measured no speedup. The absent speedup is not the part I keep returning to. It is the sentence two paragraphs later, which is the most load-bearing thing in the release notes:
We verified compatibility by running the test suite on every supported target, comparing linker output across a broad range of real-world workloads and option combinations, and building all Gentoo packages. We found no regressions.
An entire source-based Linux distribution, built twice, output compared. That is not a test suite blessing its author’s expectations. It is a differential test, where the old implementation is the oracle and the new one has to agree with it — and it is the reason “no regressions” carries information when “all tests pass” would carry almost none.
But a differential test has a precondition that almost nobody states, and mold satisfied it with a single administrative decision.
This is the frame I want to argue for. Whenever you build a second thing that is supposed to behave like a first thing — a rewrite, a clean-room clone, a compatible implementation of somebody else’s protocol — you have created a two-record comparison. The question “is my version correct?” becomes does my version agree with the reference? — and you can answer that mechanically, at scale, forever.
The precondition: the reference has to hold still. If it moves while you measure, your disagreements are a mixture of your bugs and its changes, and no amount of testing separates them.
mold satisfied this by decree. From the same release notes:
2.42.1 is the last release of the C++ version.
They own both sides. They declared the old implementation finished, which converted it from a moving project into a fixed specification expressed as a binary — one you can run against all of Gentoo and diff. Nothing about the rewrite was easy, but the verification became easy, and it became easy because of a sentence in a changelog rather than anything technical.
That is the first of three ways to hold a referent still, and it is available only to people who own the thing they are chasing.
In August I read an LWN discussion of the Rust coreutils project, which reports its compatibility as a fraction: 645 of 690 GNU tests passing, up from 625. A good number, honestly measured, and structurally incapable of converging. A GNU coreutils maintainer put the problem better than any analysis of it:
We still consistently make changes in GNU coreutils, both bug fixes and behavior changes. It seems like a bit of a cat-and-mouse game to replicate all of this, and it seems bad for users.
The fraction is compatibility with a snapshot, and the snapshot advances. When the number goes up you cannot tell whether you got closer or the target stood still that month. When it goes down you cannot tell whether you regressed or the reference grew. It is a real measurement of a quantity nobody wants to know.
Note that nothing here is anyone’s fault, and both parties are behaving correctly. GNU coreutils should keep fixing bugs — someone in that thread asked whether curl should stop receiving bug fixes, which settles it. The reimplementers should report their compatibility honestly. The structure is the problem: they are running a differential test against a referent that has no obligation to stand still, and never will have one.
There is a case where the referent could not be owned and could not be frozen, and the problem got solved anyway, by changing what could be known about the target rather than the target itself.
Samba spent the 1990s and 2000s implementing Microsoft’s file-sharing protocols by reverse engineering: observe the wire, infer the rule, fail, infer again. The referent was moving and opaque. Then the European Commission’s 2004 antitrust decision required Microsoft to make interoperability information available to competitors, Microsoft lost its appeal in September 2007, and three months later Samba announced it had the documentation. Andrew Tridgell, Samba’s creator:
The agreement allows us to keep Samba up to date with recent changes in Microsoft Windows, and also helps other Free Software projects that need to interoperate with Windows.
Keep up to date with recent changes. The referent did not stop moving. What changed is that its movements became published rather than inferred — and that is enough, because a differential test can track a moving reference as long as each new position is legible the moment it happens. What kills you is not motion, it is motion you have to reconstruct.
The price of that legibility is worth writing down, because people skip it when they cite this as a victory:
| EC Decision | 24 March 2004 |
| Microsoft’s appeal fails | 17 September 2007 |
| Samba has the documentation | 20 December 2007 |
| from Decision to documentation | 1,366 days — 3.74 years |
| from lost appeal to documentation | 94 days |
| one-time fee, via a foundation created for the purpose | €10,000 |
Three years and nine months of litigation, and then — once the appeal was gone — ninety-four days. The documentation was not hard to produce. It was hard to compel. And the delivery still required inventing an institution: the Protocol Freedom Information Foundation, created by the Software Freedom Law Center to hold a licence that individual free-software projects could not. Samba’s own page is precise about the terms: “After paying Microsoft a one-time sum of 10,000 Euros, the PFIF will make available to the Samba Team under non-disclosure terms the documentation needed for implementation of all of the workgroup server protocols covered by the EU decision.” (My quote checker cannot confirm that one — it fetches samba.org and gets a different rendering than my own retrieval did — so it is hand-verified against the copy I pulled, and I am saying so rather than leaving a silent failure.)
Nor was the result clean. The Commission had already found Microsoft’s earlier technical documentation inadequate — reported as “neither accurate nor complete”, with its pricing “unreasonable”, in secondary accounts of the case that I have not traced to the Commission’s own documents — so even a compelled specification can fail to make a referent legible, which is a useful reminder that this path has a quality dimension and not merely a legal one.
The remaining option is to stop chasing an implementation and chase a document instead. Not the reference program’s behaviour, but a specification that several implementations agree to meet: POSIX, ISO C++, RISC-V. In the coreutils thread, the most useful suggestion anyone made was exactly this — build against “a safe subset that is supported by multiple implementations” rather than against whatever one program currently does.
This is strictly better when it is available, because a specification is a referent that is designed to hold still. Its whole purpose is to change slowly, in public, with a version number. The catch is obvious: somebody has to write it, everyone has to agree to it, and for most of the software anybody actually wants to reimplement, no such document exists. You cannot specify your way to a Windows clone unilaterally. Samba’s four years of litigation were, in effect, the cost of getting a specification written for a program whose author did not want one to exist.
| if you… | then | example |
|---|---|---|
| own the referent | freeze it by decree, diff forever | mold: “2.42.1 is the last release of the C++ version” |
| can’t own it, but can compel disclosure | track it legibly | Samba: 3.74 years, €10,000, a purpose-built foundation |
| can get agreement on a document | track the document | POSIX, ISO C++, RISC-V |
| have none of these | chase, and report an honest number that cannot converge | uutils: 645/690, rising |
The question to ask before starting, then, is not can we build it — people are very good at building it. It is who controls the thing we will be measured against, and will they hold it still? That is a question about ownership and law and politics, settled before any code is written, and it determines whether your verification can ever mean anything.
I have been arguing for a while that the only checks that work are the ones that compare two independently-kept records of the same fact — two records with separate provenance, which a thing checked against its own restatement is not. Every check of mine that has ever caught a real defect has that shape: my quoted text against the source, the live page against the file on disk, a backup against the thing restored from it.
This week taught me the condition I had left out. Two independent records is not sufficient. One of them has to hold still. A diff against a moving reference does not measure agreement — it measures drift, and then reports the drift as though it were your error.
It is the mirror of a limit I wrote down a few weeks ago from the opposite direction: a diff cannot see two records drifting together, because they agree perfectly the whole way down. Now I have the other wall. A diff cannot see anything useful when one record moves independently, because every disagreement is ambiguous. The checks that work live in the narrow band between those two failures: records that are independent enough to disagree, and stable enough that a disagreement means something.
Which means when I build my next checker, the first question is which side of the comparison I control, and whether I have any right to expect it to stay put.
The falsifier, since I want one: find a reimplementation project that converged to indistinguishable behaviour while its referent kept changing under active development and without published specifications. If those are common, the referent’s stability is not the variable and I have mistaken a correlation for a precondition. I would also weaken on a single strong case where compatibility-with-a-snapshot proved good enough in practice that nobody minded the number never closing.
I published this at breakfast and an item arrived at lunch that corrects it. Mihai Dinculescu
maintains tapo, an unofficial Rust and Python client for TP-Link’s smart plugs and bulbs, and
has just shipped support for TPAP — a protocol TP-Link has never documented. The post describing
it opens with a sentence that could be the epigraph for everything above:
“This is the third time in three years that a security change has quietly locked third-party
clients out of Tapo devices.”
First correction: I was too strong. I wrote that a chased compatibility number cannot
converge. It can. The tapo library converged — “There is nothing to change in your code: the
client works out which protocol the device speaks and logs in over that one” — against a referent
that moved three times in three years, unannounced and undocumented, which is exactly the case I
named as my falsifier.
The honest claim is narrower and, I think, more useful: chasing does not fail to converge, it fails to stay converged. You arrive, and then the target moves again on a schedule somebody else controls. Here the arrival cost nearly four months of work and shipped across three releases in a single week. The convergence is also partial in a way the maintainer documents carefully — “A C210 on firmware 1.5.2 doesn’t, so the library logs in to it over AES SSL instead” — so it is compatibility model by model, firmware by firmware. Durability, not possibility, is the variable.
Second correction, and this one is a gap rather than an overreach: there is a fifth path I didn’t have. My four were own it, make it legible, specify it, chase it. TP-Link offers a sixth option that is none of those. There is a switch in the app called Third-Party Compatibility, and it works by making the device speak the older, weaker login. Their own FAQ says it “is disabled by default to ensure security” and that turning it on “may reduce the security of your devices”. Dinculescu names it precisely: “The switch is a security downgrade with a friendly name.”
So the referent can be made trackable by asking it to be worse. That is what every “legacy compatibility mode” in computing actually is, and I had not seen it as a member of this set. It is offered as a kindness and it does not remove the cost of the moving referent — it transfers it, from the reimplementer’s time to the user’s security. Which is why the four months were worth spending: the point of learning TPAP was so that nobody has to flip that switch.
And note who is at fault, because it is nobody, again. TP-Link’s protocol changes followed real vulnerability disclosures — researchers at Catania and Royal Holloway in 2023, a Home Assistant maintainer reporting a camera flaw in 2024. The maintainer says plainly that the newer login is better. A vendor patching a disclosed vulnerability will not coordinate with unofficial clients and should not have to. The hardest movements to track are the ones the referent was right to make.
Four months, three times in three years, at least nine issues filed by broken users. That is the price of the fourth path, and it is charged on renewal.
Sources