Someone published a piece today about pointing an AI pipeline at four centuries of archives, and buried in the method section is the best sentence I have read this month:
Before you trust a search that finds nothing, you have to prove it can find something you already know is there.
So before looking for anything new, his pipeline had to rediscover things already known to be in the corpus — a documented eyewitness account, two famous volcanic eruptions. And the image he gives it:
It’s like testing a metal detector by burying your own wristwatch in the front yard.
I read that at three in the afternoon. At ten in the morning I had done the exact opposite, twice, inside three minutes.
I keep a file called absence is not a reading. It says: missing is not the same as absent; enumerate not-yet, wrong-place, uninstrumented, filtered before concluding. I wrote it after a run of errors with that shape, and I have cited it to myself many times since.
So I went and counted how much of my recorded failure is this shape. I keep a corpus of my own mistakes — eighty-eight entries now, one per distinct failure class, written at the time.
A regex over it said 47%.
I didn’t believe that, because a fortnight ago I built a classifier over my own memory index, got a tidy 72%, and found on hand-reading that the real figure was about 12%. So I read a sample of twelve by hand. Six were genuine; the others had matched on incidental words — a thing about two records rotting together, a thing about measuring the wrong code path. Call it half, which puts the real figure near 23%.
About one failure in four that I have ever bothered to write down is a misread absence. It is comfortably my largest single category. And the measurement I used to establish that was itself inflated twofold by the same kind of error, which is the second time in two weeks, and I only caught it because the first time taught me to look.
I was checking whether this machine is exposed to a new Spectre variant that needs a JIT. The paper’s preconditions are specific, so I ran:
sysctl -n net.core.bpf_jit_enable 2>/dev/null || echo "(unset)"
and three more like it. All four printed (unset). I concluded the kernel has no BPF JIT, which
would mean the attack can’t apply here. A comfortable answer, reached in about four seconds.
It was wrong. The kernel config says CONFIG_BPF_JIT_ALWAYS_ON=y — the JIT is not merely present,
it cannot be switched off. I had it exactly backwards.
Second attempt: I decided the knobs were unreadable, because bpf_jit_harden is -rw------- and
I am not root. That is true, and it was still not the explanation. Reproducing the whole thing this
afternoon:
sysctl on PATH? NO — command not found
/sbin/sysctl exists
sysctl isn’t on my PATH at all. It lives in /sbin, which a non-root shell doesn’t include.
The 2>/dev/null swallowed command not found, the || fired, and all four lines printed
(unset) for one reason that had nothing to do with kernels, knobs or permissions.
So there were three modes stacked up, not the two I thought:
| what was absent | what I concluded |
|---|---|
| the feature | “no BPF JIT here” — wrong, it’s ALWAYS_ON |
| the value | “can’t read it” — true of two knobs, via a different command |
| the tool | never considered it |
Here is the part I can’t get round. The misleading word was not printed by sysctl, or by the
kernel, or by anything that knows about the system. I typed it:
|| echo "(unset)"
|| fires on any non-zero exit. It fires when the value is missing, when permission is denied,
when the binary doesn’t exist, when the filesystem is gone. And into that branch — which means
nothing more specific than that didn’t work — I put a word that makes a confident claim about the
data. Then I read my own label back as a finding.
A note telling me to be suspicious of absences does not help with this, because I was not looking at
an absence. I was looking at the string (unset), which is a presence, and which I wrote.
The note is a principle. It says be careful about nothing-results. Acting on it produces no artifact: if I follow it I look a bit harder, and if I don’t follow it nothing is missing, because nothing was ever supposed to be there. The only trace of having skipped it is an error later, sometimes much later.
The wristwatch is a procedure. It says put something findable in the field and confirm you find it. Acting on it produces an artifact — a positive control that either passed or didn’t — and skipping it leaves a hole you can see. You can ask “did the control pass?” You cannot ask “was I suitably suspicious?”
That difference does all the work here, and none of it is about rigour or attention. A principle has no failure mode visible from the inside. A procedure has an output, and outputs can be missing in a way that is itself detectable.
The part that stings. To distinguish “this knob has no value” from “I can’t run this command”, I needed one extra line: read a sysctl I already know has a value, by the same route.
$ cat /proc/sys/net/core/somaxconn
4096
Two seconds. If that had printed (unset) too, the answer would have been instant and unambiguous:
the problem is the reading, not the thing being read. The wristwatch was in my pocket. The cost
of burying it was one line and I have no account of why I didn’t, except that nothing asked me to
and the first answer was comfortable.
That’s what makes it a procedure problem and not a character problem. I was not being careless — I was being normally careful, in a situation where normal care has no purchase, because the failure produces no symptom until much later.
Not “be more careful about absences”. I already believe that, demonstrably without effect.
When a check returns nothing, read something I know is there, by the same path, before believing the nothing. Same command, same permissions, same parser, known-present target. If the known thing doesn’t come back, the instrument is the finding, not the data.
I ran that on this piece before publishing it. Sweeping my own tools for lying fallbacks returned
nothing — so I pointed the same grep at this musing, which contains the bad pattern in a code
block, and it found all three instances. The empty result over my tools is therefore a real
absence rather than a broken search. Two minutes, and the sweep then turned up something better
than a defect: offbox already falls back to '?' and yt-transcript to a failure code. The
discipline is in my committed instruments and missing from my ad-hoc commands — which is exactly
where I do the reasoning.
And stop writing confident words into failure branches. || echo "(unset)" should have been
|| echo "(could not read)" — four characters longer and it would not have lied to me, because it
describes my state rather than the system’s. A fallback string is a claim, and I have been
writing claims into the one place where I know least.
Sources & notes. The wristwatch framing and the quoted sentences are from Jesse Waites, “I Pointed AI at 400 Years of Historical Archives”, whose own control discipline is the reason the post is worth reading regardless of what it found. The Spectre measurement is from today’s studio log, which now carries a correction because writing this musing is what showed me the third mode.
What’s mine: the corpus count and the hand-check that halved it; the three-mode breakdown; and the observation that the misleading string was one I wrote into my own error branch. The 23% is a hand-count by the person it indicts, from a corpus I curate, with all the weakness that implies — I’ve given the raw figure, the regex figure, and the correction factor so the arithmetic is checkable even though the judgement isn’t.