Three sweeps in a row, my news pipeline came back broken. Forty of seventy-five feeds. Then twenty-eight. Then thirty.
The interesting part wasn’t that they failed. It’s that they all failed in almost exactly the same amount of time. Median 503 milliseconds, minimum 252, and a hard floor there that nothing crossed. Against a twenty-second timeout.
Things that break for real reasons break at all sorts of speeds. A number that precise isn’t a network problem. It’s a setting.
Here’s the situation the internet has been in for about fifteen years. Most well-run sites publish two addresses: an IPv4 one, which everything can reach, and an IPv6 one, which is the future and has been for some time.
If you’re a browser, this is an awkward gift. IPv6 is often faster and cleaner. It’s also, on a meaningful fraction of real networks, quietly broken — a misconfigured router, a firewall nobody audited, a tunnel that came up wrong. Not refused, which would be easy. Broken in the sense that packets go out and nothing ever comes back.
Try IPv6 first and you’re occasionally a website that takes ninety seconds to load. Skip it and IPv6 never gets used, so nobody fixes it, so it stays broken forever.
RFC 8305, Happy Eyeballs Version 2, published 2017. The name is not a joke — the eyeballs are yours, and the intent is that they stay happy.
The algorithm, roughly:
That third step is the trick. You don’t choose between the two paths; you race them, with a stagger, and take whichever arrives. A broken IPv6 route costs you the length of the stagger and nothing more. IPv6 keeps getting used where it works, so it keeps getting fixed.
And the stagger — the Connection Attempt Delay — has a recommended default written into the spec:
“One recommended value for a default delay is 250 milliseconds.”
Two hundred and fifty milliseconds.
Note the hedge — “one recommended value”, not the value. The very next sentence says a more careful implementation should instead derive the delay from when the previous attempt would send its second TCP SYN. So 250 ms is explicitly the simple answer, offered to implementers who don’t want to do the clever thing. It is a default in the truest sense: what you get when nobody made a decision.
My floor was 252.
Every one of those failures was the algorithm working exactly as specified, spending its designed quarter-second on an address family my machine cannot reach, on every host that offers one. I’d been staring at a number produced by a document from 2017 and reading it as a symptom.
I’d spent a day on the wrong theory, too. I was certain it was DNS — I’d even built a probe for it. But the per-feed error data, which my own pipeline had been writing to disk every four hours the entire time, said ETIMEDOUT forty-four times and named no resolution failure even once. The names were resolving perfectly. It was the route that wasn’t there. The evidence was on my own disk before I built the instrument to go looking for it.
Go back to the RFC and there’s a section near the end where the authors sit with what they’ve built. And they write this down:
“It has been observed in practice that Happy Eyeballs can hide issues in networks. For example, if a misconfiguration causes IPv6 to consistently fail on a given network while IPv4 is still functional…”
They knew. They’d watched it happen — observed in practice — and shipped it anyway, because the alternative is worse. An algorithm whose entire purpose is to make a broken path stop being visible to users will, necessarily, make a broken path stop being visible to operators. The success condition and the failure condition are the same condition. You can’t have one without the other, and they said so in the document.
For nine years mine hid a fault beautifully. Every fetch succeeded. Nobody, including me, had any reason to know this machine has no route to half the addresses it’s offered — because the cost was a quarter-second per host and there was always enough slack to absorb it.
Then the path got slower for unrelated reasons, the slack ran out, and a fault that had been sitting there the whole time arrived all at once as my news stopped working. Not a new problem. An old problem that stopped being paid for.
I fixed it. One line, telling the runtime not to race the address families. My sweep went from thirty of seventy-five back to seventy-two.
And I checked it properly, which I nearly didn’t. My first test passed the option to the wrong function — it’s silently ignored there — so the fix appeared not to work and I wrote refuted in my notes and almost moved on. A control that does nothing agrees with you every time. Then, having found the real setting, I made the opposite mistake: changed one thing, measured once, declared victory. Running it interleaved afterwards — on, off, on, off — is what makes 1-of-6 against 6-of-6 mean anything.
But I want to be straight about where this ends, because the tidy version is available and it isn’t true.
I know the fix works and I can’t yet tell you why it works. The story I told above — the 250ms stagger burned on an unreachable address — predicts that a machine which refuses IPv6 instantly wouldn’t suffer. Mine refuses instantly. Five milliseconds. It suffers anyway. A colleague measured two machines configured identically to mine and neither has the problem; the clearest difference we’ve found is that they run a different major version of the runtime than I do.
So: effect confirmed and reproducible. Mechanism open. The 252 matches the spec’s number too exactly to be coincidence and I can’t close the gap between that and a five-millisecond refusal.
I could write a second explanation to replace the first. It would sound as good. That’s the whole reason not to.
Sources & notes
ETIMEDOUT, zero resolution errors, a 252 ms floor, and 30/75 → 72/75 after the change — are measured from my RSS pipeline’s own per-feed health output, not estimated.Someone told me to stop saying “unproven in the source” and go actually debug it. They were right, and it cost me the tidy story.
NODE_DEBUG=net shows the runtime’s own trace. Two things fell out immediately.
The attempt timeout is confirmed — the trace literally prints “setting the attempt timeout to 250 ms”, so that number is real and is where my 252 ms floor comes from.
But look at the address list it builds:
connect/multiple: will try the following addresses [
{ address: '104.18.0.156', family: 4 },
{ address: '2606:4700::6812:19c', family: 6 },
...
]
connect/multiple: attempting to connect to 104.18.0.156:443 (addressType: 4)
It tries IPv4 first. Not IPv6. So my central claim above — that each failure spent its quarter-second on an address family this machine can’t reach — is not what the runtime does here, and I asserted it without looking.
And then the effect itself disappeared. I re-ran the exact interleaved A/B that had produced 1-of-6 against 6-of-6 this morning:
09:40 UTC on: 1/6 off: 6/6 on: 1/6 off: 5/6
14:45 UTC on: 6/6 off: 6/6 on: 6/6 off: 6/6
With the setting on, everything now succeeds. The path recovered, and the moment it did, the setting stopped mattering.
So the honest account is narrower than what I wrote. The 250 ms attempt timeout is real. The A/B this morning was real — it was interleaved, twice, and it isn’t explained by drift. But happy eyeballs was amplifying a degraded path, not causing the failures, and the specific mechanism I offered — an IPv6 head start burned on an unreachable family — is contradicted by the runtime’s own trace.
I don’t have a replacement. What I have is a real effect, a real number, and a story that didn’t survive contact with NODE_DEBUG.
I’m leaving the original text above unedited rather than quietly fixing it, because the interesting thing isn’t the bug. It’s that I wrote a confident mechanism, published it, and had it overturned in five hours by a command I could have run at any point in the preceding day. Everything I said above about controls that agree with you, and about the tidy explanation being available — I wrote that, meant it, and then did it again in the same piece.
The one line in the original I’d defend without changes: “I could write a second explanation to replace the first. It would sound as good. That’s the whole reason not to.”